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IMAGING METHODS AND DEVICES 



rOQOn This application is a S 371 from PCT/US2005/Q07585 filed March 

7> 2005> which claims priority benefit of provisional patent application Serial No. 
60/550.614 filed March 6> 2004. which is incorporated bv reference in its entirety and is a 
continuation-in- part of application Serial No. 10/832.684 filed April 26. 2004. each of 
which is incorporated by reference in its entirety. 

FIELD OF THE INVENTION 

fQQQj4r00021 The present inyention relates generally to methods and devices for 
hyper-spectral imaging, more particularly to MEMS-modulated-aperture imaging 
spectrograph systems and methods. 

BACKGROUND OF THE INVENTION 

[00021 [00031 Imagers employ either a two-dimensional (2D) multichannel 
detector array or a single element detector. Imagers using a 2D detector array measure 
the intensity distribution of all spatial resolution elements simultaneously during the 
entire period of data acquisition. Imagers using a single detector require that the 
individual spatial resolution elements be measured consecutively via a raster scan so that 
each one is observed for a small fraction of the period of data acquisition. Prior art 
imagers using a plurality of detectors at the image plane can exhibit serious signal-to- 
noise ratio problems. Prior art imagers using a single element detector can exhibit more 
serious signal-to-noise ratio problems. Signal-to-noise ratio problems limit the utility of 
imagers applied to chemical imaging applications where subtle differences between a 
sample's constituents become important. 

fOQQ^[00041 Spectrometers are commonly used to analyze the chemical 
composition of samples by determining the absorption or attenuation of certain 
wavelengths of electromagnetic radiation by the sample or samples. Because it is 
typically necessary to analyze the absorption characteristics of more than one wavelength 
of radiation to identify a compound, and because each wavelength must be separately 
detected to distinguish the wavelengths, prior art spectrometers utilize a plurality of 
detectors, have a moving grating, or use a set of filter elements. However, the use of a 
plurality of detectors or the use of a macro moving grating has signal-to-noise limitations. 
The signal-to-noise ratio largely dictates the ability of the spectrometer to analyze with 
accuracy all of the constituents of a sample, especially when some of the constituents of 
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the sample account for an extremely small proportion of the sample. There is, therefore, a 
need for imagers and spectrometers with improved signal-to-noise ratios. 

ffiQQ44r00051 Prior art variable band pass filter spectrometers, variable band 
reject filter spectrometers, variable multiple band pass filter spectrometers or variable 
multiple band reject filter spectrometers typically employ a multitude of filters that 
require macro moving parts or other physical manipulation in order to switch between 
individual filter elements or sets of filter elements for each measurement. Each filter 
element employed can be very expensive, difficult to manufacture and all are 
permanently set at the time of manufacture in the wavelengths (bands) of radiation that 
they pass or reject. Physical human handling of the filter elements can damage them and 
it is time consuming to change filter elements. There is, therefore, a need for variable 
band pass filter spectrometers, variable band reject filter spectrometers, variable multiple 
band pass filter spectrometers or variable multiple band reject filter spectrometers without 
a requirement for discrete (individual) filter elements that have permanently set band pass 
or band reject properties. There is also a need for variable band pass filter spectrometers, 
variable band reject filter spectrometers, variable multiple band pass filter spectrometers 
or variable multiple band reject filter spectrometers to be able to change the filters 
corresponding to the bands of radiation that are passed or rejected rapidly, without macro 
moving parts and without human interaction. 

r0005i r00061 in several practical applications it is required that an object be 
irradiated with radiation having particularly shaped spectrum, hi the simplest case when 
only a few spectrum lines (or bands) are necessary, one can use a combination of 
corresponding sources, each centered near a required spectrum band. Clearly, however, 
this approach does not work in a more general case, and therefore it is desirable to have a 
controllable radiation source capable of providing arbitrary spectrum shapes and 
intensities. Several types of prior art devices are known that are capable of providing 
controllable radiation. Earlier prior art devices primarily relied upon various "masking" 
techniques, such as electronically alterable masks interposed in the optical pathway 
between a light source and a detector. More recent prior art devices use a combination of 
two or more light-emitting diodes (LEDs) as radiation sources, hi such cases, an array of 
LEDs or light-emitting lasers is configured for activation using a particular encoding 
pattern, and can be used as a controllable light source. A disadvantage of these systems is 
that they rely on an array of different LED elements (or lasers), each operating in a 
different, relatively narrow spectrum band. In addition, there are technological problems 
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associated with having an array of discrete radiation elements with different 
characteristics. Accordingly, there is a need for a controllable radiation source, where 
virtually arbitrary spectrum shape and characteristics can be designed, and where 
disadvantages associated with the prior art are obviated. Further, it is desirable not only 
to shape the spectrum of the radiation source, but also encode its components differently, 
which feature can be used to readily perform several signal processing functions useful in 
a number of practical applications. The phrase "a spectmm shape" in this disclosure 
refers not to a mathematical abstraction but rather to configurable spectrum shapes having 
range(s) and resolution necessarily limited by practical considerations. 

[0006] [0007] In addition to the signal-to-noise issues discussed above, one can 
consider the tradeoff between signal-to-noise and, for example, one or more of the 
following resources: system cost, time to measure a scene, and inter-pixel calibration. 
Thus, in certain prior art systems, a single sensor system may cost less to produce, but 
will take longer to fully measure an object under study. In prior art multi-sensor systems, 
one often encounters a problem in which the different sensor elements have different 
response characteristics, and it is necessary to add components to the system to calibrate 
for this. It is desirable to have a system with which one gains the lower-cost, better 
signal-to-noise, and automatic inter-pixel calibration advantages of a single-sensor system 
while not suffering all of the time loss usually associated with using single sensors. 

[0007] [0008] The conventional spectral imaging systems can be generally 
categorized into three types: so-called pushbroom imagers, filter scanned imagers, and 
modulated focal plane array systems. FIG. 60 depicts a typical pushbroom spectral 
imaging system. In pushbroom imagers, a scene 8000 is imaged onto the entrance 
aperture of an imaging spectrograph containing a slit mask 8010. One column of spatial 
resolution elements defined by rows of detectors at the focal plan of an imaging 
spectrograph £ire spectrally imaged through the slit for each fi-ame of data captured by the 
focal plane. The light fi-om the slit is spectrally dispersed 8020 along a direction 
perpendicular to the direction of the slit, generating a rectangular image 8030 which 
varies spatially along an axis parallel to the slit, and spectrally along the orthogonal axis. 
A focal plane array is typically used to capture the rectangular image. To image a scene, 
the slit is translated such that it moves incrementally across the scene, with the 
pushbroom system collecting one image for each position of the slit. In this way, the 
pushbroom system collects a so-called hyper-spectral image cube of the scene. Note that 
it is standard in the art to refer to such a dataset as an image cube or a data cube or a 
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datacube. This use of "cube" is imprecise, in that the underlying dataset may be 
rectangular (i.e. not necessarily of the same size in all 3 dimensions). The scanning of the 
slit across the scene may be accomplished by moving the imaging system, moving the 
scene or an object of interest, or optical scanning (e.g. with a moving (macro) mirror). 

fQ0Q84rQ0091 However, the slit must be very narrow in the pushbroom system to 
achieve the desired spectral resolution. At any given time, a narrow slit only accepts a 
very small fraction of the light from the entire scene, thus making the hyper-spectral 
imaging systems much less sensitive than the conventional imaging systems. That is, the 
pushbroom system will either acquire data of low signal to noise ratio or take longer to 
acquire the data. In the latter case, the pushbroom system is limited to imaging scenes 
that do not change over time. For changing scenes, the pushbroom system will suffer 
from artifacts. 

fQOQ94rOO 1 01 A dditional! V, the scanning must be mechanically precise with the 
pushbroom system. Any scanning errors due to mechanical imprecision in the 
pushbroom system will result in a distorted data set. Further, the need to move the 
camera or the object of interest in certain pushbroom system is undesirable for a variety 
of reasons, including but not limited to the cor^responding costs and complexities of 
associated additional system components, and the limitation that the system may only be 
used in contexts where it is possible to move the object or the camera in a controlled 
manner. 

["0Q10] ["0011] In filter scanned imagers, an ordinary broadband camera images a 
scene of interest, but a tunable filter is inserted somewhere in the optical path between the 
scene and the camera. The filter may be a liquid crystal tunable filter such as a CRI 
VariSpec LCTF (see, http://www.cri-inc.com/filesA/^ariSpec_Brochure.pdf ), or any 
similar device that transmits a narrowband of wavelengths at any given time, with the 
center wavelength of the band tunable in time. FIG. 316j- depicts a typical liquid crystal 
tunable filter scanned spectral imaging system 3000. Light from the object 3010 enters 
the coUimating lens system 3020 where it encounters the liquid crystal tunable filter 
system 3030 and then propagates onto the focusing lens system 3040 where the object is 
focused onto the focal plane 3050. 

fOOj4^[0012] However, the bandpass of the filter in the filter scanned imager 
must be very narrow to achieve the desired spectral resolution. Like the slit in the 
pushbroom systems, at any given time, a narrow bandpass only accepts a very small 
fraction of the light from the entire scene, thus making the hyper-spectral imaging 
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systems much less sensitive than ordinary imaging systems. That is the fiher scanning 
system will either acquire data of low signal to noise ratio (SNR) or take longer to acquire 
the data. In the latter case, the filter scanning system is only limited to imaging scenes 
that do not change over time. For changing scenes, the filter scanning system will suffer 
fi-om artifacts. 

fOQl^rOOl 31 Multiplexed focal plane array spectral imaging systems, such as 
Fourier transform interferometric imaging systems, typically employ an imaging optical 
system, the output of which is passed through an interferometric assembly, and then 
imaged onto a focal plane array. As the interferometer is scanned, a multiplexed spectral 
image is acquired. FIG. depicts a conventional scanning multiplexed focal plane 

array spectral imaging system, such as a Fourier transform focal plane array spectral 
imaging system 4000. The object source 4010 is collected by image grade coUimating 
optics 4020 where it is coUimated onto a beam splitter 4030 that splits the energy 50/50 to 
stationary mirror 4040 and to a moving mirror 4050. This is then recombined at the 
beamsplitter 4030 and propagates onto the focusing optics 4060 and is re-imaged onto the 
focal plane 4070. 

ffiQ4^[0014] However, in Fourier transform interferometric imaging and other 
similar systems, distortions and extreme system sensitivities can result from passing the 
light of an imaging system through an interferometer. This leads to the distorted data, as 
well as system complexity and extreme sensitivity of motion. 

[00 1 4 ] [00 15] Each of the foregoing prior art systems scans through a full hyper- 
spectral datacube. However, the output of such measurement fi-om such prior art system 
is generally an input to an algorithm that processes each pixel, and produces an answer 
consisting one or a few nimibers per pixel. For example, these could be the output of a 
set of inner products, as is standard in the art of chemometrics. Since hyper-spectral 
datacubes contain a large amount of data and the answer consists of a smaller amount of 
data, it is desirable to find a method for directly measuring such answer. In other words, 
it is desirable to find a method for enabling adaptive measurements of spectral image 
parameters. 

fO&j-^[00 1 6] Conventional Hadamard transform spectroscopy (HTS), Hadamard 
transform imager (HTI), and Hadamard transform spectral imager (HTSI) overcome only 
some of the limitations and problems described herein. Hadamard optical systems utilize 
spatially encoded apertures that can be employed either at the entrance aperture of an 
optical system, the exit aperture or both. They have the common attribute that they 
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encode the available aperture spatially where the spatial resolution elements that make up 
the encodement dictate the spectral, spatio-spectral or spatial resolution elements that 
propagate through the optical system including diffractive optical elements and on to the 
sensor or exit aperture. These masks have some spatial extent that places special 
requirements on the optics of the system. As the encodement mask grows either by 
longer length encodements with fixed sub-apertures or as the sub-aperture dimension 
grows for a fixed encodement length, the spatial resolution elements making up the sub- 
apertures in the encodement mask depart from the optical axis. When the resolution 
elements depart from the optical axis or paraxial condition it is desirable to employ optics 
that can image the off axis resolution elements without inducing excessive aberrations 
that degrade the performance or cripple the advantages gained by Hadaniard transform 
(HT) multiplexing. 

fQ&j-6^[00 1 71 Tvpicallv the optical path for conventional monochromators begins 
with a source that is focused onto an aperture plane that has a large aspect ratio aperture 
known as a slit. This slit is often very small in extent in the dispersion plane compared to 
the other extent in the spatial plane. However, it is not required that this aspect ratio be 
large. If the aspect ratio is close to 1 then simple spherical optics can be employed that 
perform well as long as the departure from the optical axis is kept to a minimum. 
However, most monochromators have a large aspect ratio in order to increase the 
opportunity to maximize throughput, and detectors must be able to "see" the large extent 
of the slit aperture. The light entering the slit aperture is then dispersed and focused onto 
an exit slit aperture. Monochromators generally perform well on the optical axis and do 
not typically employ optics that can manage rays that depart from the optical axis in the 
plane of dispersion as required by HT multiplexing instruments. The optical system 
generally utilizes optical performance attributes normally found only in imaging and 
spectral imaging systems to employ encoding techniques. This requirement is driven by 
the extent of the encoding mask. The extent of the encoding mask is governed by the 
diffraction limit of the wavelengths within the bandpass, the encodement length N and the 
attributes of the optical system. 

fO&j^fOQ181 In a conventional dispersive spectrometer the radiation from a 
source is collected and separated into it's individual spectral resolution elements by a 
spectral separator such as a diffraction grating or prism and then is collected and focused 
for spatial presentation on a focal plane. The dispersive spectrometer uses a single exit 
slit to select one spectral resolution element of N spectral resolution elements for 
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measurement by the detector. The Hadamard transform spectrometer (HTS) uses an array 
of slits (i.e. a mask) at the focal plane to select one more than half, (N+l)/2, of the 
spectral resolution elements at the focal plane for measurement by the detection system. 
The optical challenge to effect an HT multiplexing spectrometer is to collect all of the 
spatially distributed individual band pass images of the entrance slit and transfer them to 
as small detector as possible. It is desirable to keep the area of the detector at a minimum 
as the noise of many detectors increases with the square of the area. If the optics are able 
to illuminate a single detector element with all of the available light impinging upon the 
focal plane containing the spatially distributed images of the slit for each of the N band 
pass resolution elements, a multitude of spectral resolution elements can be measured 
simultaneously using a single detector element. This arrangement results in a 
multiplexing spectrometer. The recovery of N spectral resolution elements requires 
measuring the detector response for N different encodements of (N+l)/2 open mask 
elements. The raw data is recorded as the detector response versus encodement number 
and is called an encodegram. Hadamard transformation of the encodegram yields the 
spectrum. 

fQ&j-84r00191 The Hadamard transform instruments developed in the 1960s and 
1970s employed moving masks. Significant problems such as misalignment and jamming 
associated with a moving mask led to a reputation of poor reliability and contributed to a 
dormant period in the development of Hadamard transform spectrometer (HTS) and 
Hadamard transform imager (HTI). Interest was rekindled in the 1980s using stationary 
Hadamard encoding mask based on liquid crystal (LC) technology. The first generation 
ID stationary Hadamard encoding mask was a cholesteric LC with N = 127 mask 
elements and used polarization as its operating phenomenon. Two parallel polarizers and 
rotation or lack of rotation of the polarized radiation generated the opaque and transparent 
states, respectively. The second generation ID stationary Hadamard encoding mask was 
fabricated using a polymer dispersed liquid crystal (PDLC) material with N = 255 mask 
elements and used light scattering as its operating phenomenon. The PDLC contained LC 
droplets dispersed in a polymer matrix whose index of refi^action matched the index of 
refi-action in one direction in the birefiingent LC droplet. Alignment of the LC droplets 
optical axis under an applied voltage removed discontinuities in index of refi-action at the 
polymer matrix/LC interface to generate a good transparent state while random 
orientation of LC droplets in the polymer matrix generated the opaque state from light 
scattering by the discontinuities in index of refraction at the polymer matrix/LC droplet 
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interface. A 2D stationary Hadamard encoding mask was also based on LC technology. 
A fero-electric liquid crystal (FLC) positioned between a pair of polarizers with 
perpendicular orientation operated as an electro-optic half-wave plate when a + value of 
applied voltage rotated the plane of polarization by 90 degrees to produce the transparent 
state and a - value of applied voltage left the plane of polarization unaltered to produce 
the opaque state. 

fO&j-94f0020] Development based on stationary Hadamard encoding masks 
continued in the 1990s and a 2D moving Hadamard encoding mask was also fabricated 
and used to perform imaging in the near-infrared and mid-infrared spectral regions. Note 
that the mid-infrared spectral region is not generally accessible via Hadamard encoding 
masks based on LC technology since any LC material generally has strong absorption 
bands in the mid-infrared spectral region. A stationary Hadamard encoding mask of 
available for the visible and near-infrared spectral regions is the digital micro-mirror 
device (DMD), a device based on micro-optoelectromechanical systems (MOEMS) 
technology and developed by Texas histruments for projector display applications. One 
DMD format incorporates 508,800 micro-mirrors in a 848 column by 600 row array that 
is 14.4 nmi wide by 10.2 mm high. Each individual micro-mirror is 16 microns square 
and adjacent micro-mirrors are separated by a 1 |xm gap. The micro-mirrors are 
individually addressable and rotatable by +10 or -10 degrees about the diagonal axis to 
produce binary "on" and "off states. The on state has Tt determined by the mirror 
reflectivity and approaches 1 while the off state approaches To = 0. However, the ideal 
condition of on and off is not realized due to diffraction of the light off of the small and 
periodic features of the micro-mirror device. 

fOQ3O4r002l1 The DMD is an array of spatial resolution elements that can be 
selected as groups of super-resolution elements or as individual resolution elements 
consisting of a single micro-mirror. The DMD resolution elements are disposed as 
spectral resolution elements in the spectrometer with the columns attributed to the 
frequency or wavelength dimension and the rows attributed to the slit height dimension. 
The DMD resolution elements are utilized as spatio-spectral resolution elements in the 
imaging spectrograph with the columns as the frequency or wavelength dimension and 
the rows as a vertical spatial dimension with the horizontal spatial dimension being 
accessed, if desired, by translating the sample relative to the imaging spectrograph. The 
DMD resolution elements are spatial resolution elements in the imager with the columns 
for the horizontal dimension and the rows for the vertical dimension and the frequency or 
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wavelength dimension provided by other instrumentation. If a photo-acoustic detection 
system to is present then the depth dimension of the sample can also be accessed by 
changing the modulation frequency used in the photo-acoustic detection system. 

ffiQ2j4r00221 T he notable features of HTS, HTI and HTSI are: a multiplexing 
technique using a single-element detector; uses a Hadamard encoding mask (multi-slit 
array) in the focal plane; sends one more than half the resolution elements to the single- 
element detector in an encodement; uses a number of encodements equal to the number of 
resolution elements desired and the number of mask elements (pixels) in the stationary 
Hadamard encoding mask (a moving Hadamard encoding mask has 2N - 1 mask 
elements); each encodement contains a different combination of one more than half the 
resolution elements; the primary data is the encodegram, a record of detector response 
versus encodement number; and uses a FHT of the encodegram to decode the 
encodegram and generate the spectrum or image. Additionally, HTS is a dispersive 
technique using a single-element detector. 

fOQ234r00231 However even considering these notable features of HTS, HTI and 
HTSI, there is still a need for a multiplexed spectral imaging system that can accomplish 
its multiplexing and scene scanning without macro-moving parts while maintaining the 
advantages of a DMD-based Hadamard transform spectral image collection system. 
Additionally, the present invention proceeds upon the desirability of providing such 
system that can be digitally controlled, enabling adaptive measurement of spectral image 
parameters. 

fOO£^rQ0241 Beginning with Martin Harwit's pioneering design, U.S. Pat. No. 
3,720,469, extending M. Golay's fundamental work of the 1940's to Hadamard transform 
imaging spectroscopy, many novel spectrographs have been conceived that apply fixed 
optical meisks mechanically scanned with respect to the object being imaged. We note 
two relevant recent examples that employ novel mask designs and coded apertures 
schemes similar to the present invention, but different in that they require mechanical 
translation of a fixed mask. 

fQ0344[0025] Stephen Mende, in U.S. Pat. No. 5.627,639, discloses various 
coded aperture methods for imaging spectrographs. He employs novel Hadamard mask 
including the possibility of a spatial light modulator (LCD) but still requires the scene 
being imaged to be translated with respect to the mask. 

fQ02^r0026] Whereas, in "HADAMARD imaging spectrometer with microslit 
matrix", Rainer Riesenberg; Ulrich Dillner; Proc. SPIE Vol. 3753, p. 203-213 (Oct 1999), 
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the authors describe a fixed novel MEMS mask consisting of tiny mirror pixels arranged 
in a coded pattern. However, the mirrors are fixed and, again, the mask is placed at the 
entrance of an imaging spectrograph and mechanically scanned across the aperture in 
order to Hadamard encode the input. 

fQQ264rOQ271 Given the recent developments in micro-mirror array technology, 
pioneered by Texas Instruments^^, the ability to digitally control the position of the 
individual mirrors has enabled several novel spectrograph designs that no longer require 
scanning in the conventional sense. For example, in "Characterization of a digital 
micromirror device for use as an optical mask in imaging and spectroscopy", Kevin J. 
Kearney; Zoran Ninkov; Proc. SPIE Vol. 3292, p. 81-92 (Apr. 1998), the authors propose 
a design for a Multi-Object-Spectrograph (MOS). Their system involves an imaging 
spectrograph with a micro-mirror array placed at the entrance image plane. The mirrors 
are adaptively used to select a single object per line of the array but at any location within 
that line, thus allowing the spectrograph to query multiple objects in the scene no longer 
necessarily vertically aligned as required by conventional line-scanning spectrographs. 
However, in a sense, this system uses a spatial light modulator to avoid the multiplexing 
taken advantage of in the present invention that intentionally records overlapping spectra 
on the focal plane array. 

OBJECT AND SUMMARY OF THE INVENTION 

ffi037^[0028] A n object of the present invention is provide a hyper-spectral 
imaging system which uses a micro-mirror array instead of the traditional entrance slit of 
a conventional spectral measurement system. In accordance with an embodiment of the 
present invention, the detector of the hyper-spectral imaging system as aforesaid 
comprises a camera, charge coupled device (CCD), focal plane array or other similar 
device. 

fQQ284rQ0291 In accordance with an embodiment of the present invention, a 
scene or object of interest is imaged onto a spatial light modulator. The spatial light 
modulator is used to pass or reject the spatial resolution elements of the image. The 
selected spatial resolution elements propagate through an imaging spectrograph system 
and are spectrally imaged onto a focal plane array sensor. 

fQQ294r0030] In accordance with an embodiment of the present invention, the 
hyper-spectral imaging system is capable of multiplexed measurements, including 
Hadamard hyper-spectral imaging and an adaptive multiplexing. It is appreciated that 
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applications of the present invention in spectrometry and imaging are not limited to HT 
techniques since the information corresponding to any micro-mirror in the DMD can be 
included in or excluded from any measurement as desired by the investigator or operator. 
In accordance with an aspect of the present invention, an instrument comprising no 
moving parts other than the micro-mirrors in the DMD provides a compact and robust 
instrument for operation in rugged or potentially hostile environments ranging from 
process control to outer space. 

fOQ^O4rQ03 1 1 Advantages of the present invention include, but not limited to, 
extended of hours of operability, operable capabilities beyond the conventional single slit 
spectral imaging systems, improved signal to noise ratios, increased sensitivity for 
chemistry and features of interest in hyper-spectral imagery and spectral data, faster 
information delivery, smaller bandwidth for telemetry of data, and post processing 
reduction of data. 

fflQ^44r00321 In accordance with an embodiment of the present invention, the 
hyper-spectral imaging system is a DMD-modulated-aperture imaging spectrograph 
system operable in the near-infrared or a Near-infrared Spectral Target Identification 
System (NSTIS). The NSTIS comprises fore optics for imaging a scene of interest, a 
micro-mirror array spatial light modulator (SLM or a digital micro-mirror device 
(DMD)), a diffraction grating and an infrared camera, a device for transferring images 
between these optical components, and electronics to drive the SLM. 

[0032] [0033] It is intended that the devices and methods in this application in 
general are capable of operating in various ranges of electromagnetic radiation, including 
the ultraviolet, visible, infrared, and microwave spectrum portions. Further, it will be 
appreciated by those of skill in the art of signal processing, be it acoustic, electric, 
magnetic, etc., that the devices and techniques disclosed herein for optical signal 
processing can be applied in a straightforward way to those other signals as well. 

BRIEF DESCRIPTION OF THE DRAWINGS 

fOO^^r0034] The present invention will be understood and appreciated more 
fully from the following detailed description, taken in conjunction with the drawings in 
which: 

fOOM^( "0035] FIGs. 1 A and IB are schematic diagrams illustrating a spectrometer 
constructed in accordance with two embodiments of the invention; 
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fQQ^r0036] FIG. 2 is a plan view of a micro-mirror array used in the present 
invention; 

fQQ^r00371 F IG. 3 is a schematic diagram of two micro-mirrors illustrating the 
modulations of the mirrors of the micro-mirror device of FIG. 2; 

fQQ^r00381 FIG. 4 is a graph illustrating an output signal of the spectrometer 
when used to analyze the composition of a sample; 

fOQ^rOQ391 FIG. 5 is a graph illustrating an output signal of the imager when 
used for imaging purposes; 

fQ0^94rQ0401 FIG. 6 is a schematic diagram illustrating an imager constructed in 
accordance with a preferred embodiment of the invention; FIG. 6A illustrates spatio- 
spectral distribution of a DMA, where individual elements can be modulated; 

fQ04Qir004 1 1 F IG. 7 is an illustration of the input to the DMA Filter 
Spectrometer and its use to pass or reject wavelength of radiation specific to constituents 
in a sample; 

fQQ444r0042] F IG. 8 illustrates the design of a band pass filter in accordance with 
the present invention (top portion) and the profile of the radiation passing through the 
filter (bottom portion); 

fQ0424r00431 FIG. 9 illustrates the design of multi-modal band-pass or band- 
reject filters with corresponding intensity plots, in accordance with the present invention; 

fQQ4^[0044] FIG. 1 0 illustrates the means for the intensity variation of a spectral 
filter built in accordance with this invention; 

fOQ444r00451 FIGs. 11-14 illustrate alternative embodiments of a modulating 
spectrometer in accordance with this invention; FIGs. 1 lA and 1 IB show embodiments in 
which the DMA is replaced with concave mirrors; FIG. 12 illustrates an embodiment of a 
complete modulating spectrometer in which the DMA element is replaced by the concave 
mirrors of FIG. 11. Figure 13 illustrates a modulating lens spectrometer using lenses 
instead of DMA, and a "barber pole" arrangement of mirrors to implement variable 
modulation. FIG. 14. illustrates a "barber pole" modulator arrangement; 

fQQ4^r00461 FIGs. 15 and 16 illustrate an embodiment of this invention in 
which one or more light sources provide several modulated spectral bands using a fiber 
optic bundle; 

ffiO464r0047] FIG. 1 7 illustrates in diagram form an apparatus using controllable 
radiation source; 
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fQ04?^[Q0481 FIGs. 1 8 A and 1 SB illustrate in a diagram form an optical synapse 
processing unit (OSPU) used as a processing element in accordance with the present 
invention; 

fQQ484rOQ491 FIG. 19 illustrates in a diagram form the design of a spectrograph 
using OSPU; 

fQQ494r00501 FIG. 20 illustrates in a diagram form an embodiment of a tunable 
light source; 

r0050T r00511 fig. 21 illustrates in a diagram form an embodiment of the spectral 
imaging device, which is built using two OSPUs; 

ffiO#j4r00521 FIGs. 22 and 23 illustrate different devices built using OSPUs; 

fOQS2ir00531 FIGs. 24-26 are flow charts of various scans used in accordance 
with the present invention. Specifically, FIG. 24 is a flow chart of a raster-scan used in 
one embodiment of the present invention; FIG. 25 is a flowchart of a Walsh-Hadamard 
scan used in accordance with another embodiment of the invention. FIG. 26 is a 
flowchart of a multi-scale scan, used in a different embodiment; FIG. 26A illustrates a 
multi-scale tracking algorithm in a preferred embodiment of the present invention; 

r0053i r00541 FIG. 27 is a block diagram of a spectrometer with two detectors; 

fQQ§4^r00551 FIG. 28 illustrates a Walsh packet library of patterns for N = 8. 

fOO§^rQ056] FIG. 29 is a generalized block diagram of hyper-spectral 
processing in accordance with the invention; 

rQ056i r00571 FIG. 30 illustrates the difference in two spectral components (red 
and green) of a data cube produced by imaging the same object in different spectral 
bands; 

fQQ^ytf0058] F IG. 31 illustrates hypor spectral imaging from airborn e cam e rag 
conventional liquid crvstal tunable filter scanned spectral imaging svstem : 

fQOS84r00591 FIG. 32 is an illustration of a h>per - spectral imag e of human 
ski ttillustrates a conventional scanning multiplexed spectral imaging svstem. such as a 
Fourier transform focal plane arrav spectral imaging svstem : 

fQQ^r00601 F IGs. 33A-E illustrate different embodiments of an imaging 
spectrograph used in accordance with this invention in de-dispersive mode; 

r006Q] [0061] FIG. 34 shows an axial and a cross-sectional views of a fiber optic 
assembly; 
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ffiO^r00621 FIG. 35 shows a physical arrangement of the fiber optic cable, 
detector and the slit; FIG. 36 illustrates a fiber optic surface contact probe head abutting 
tissue to be examined; 

ffiQ6a;rOQ631 F IGs. 37A and 37 B illustrate a fiber optic c-Probe for pierced ears 
that can be used for medical monitoring applications in accordance with the present 
invention; 

fOQ6^r00641 FIGs. 38A, 38B and 38C illustrate different configurations of a 
hyper-spectral adaptive wavelength advanced illuminating imaging spectrograph 
(HAWAIIS) in accordance with this invention; 

fOQ644r00651 FIG. 39 illustrates a DMA search by splitting the scene; 

fOQ6^[Q0661 FIG. 40 illustrates wheat spectra data (training) and wavelet 
spectrum in an example of determining protein content in wheat; 

fQ066;r00671 FIG. 11 illustratos FIGs. 41-1 and 41-2 illustrate the top 10 wavelet 
packets in local regression basis selected using 50 training samples in the example of 
FIG. 40; FIG 42 is a scatter plot of protein content (test data) vs. correlation with top 
wavelet packet; Fig 43 illustrates PLS regression of protein content of test data; 

fflQ674r00681 FIG. 44 illustrates the advantage of DNA-based Hadamard 
Spectroscopy used in accordance with the present invention over the regular raster scan; 

fQ06^[00691 FIGs. 45-49(A-D) illustrate hyperspectral processing in accordance 
with the present invention; 

fQ0694r00701 FIG. 50 is a block diagram of an exemplary NSTIS in accordance 
with an embodiment of the present invention; 

fOQ?Q^[0071] FIG. 51 illustrates an example of the tilted object plane imaged 
onto a resultant tilted image plane using a simple image relay lens system in accordance 
with an embodiment of the present invention; 

fOO^Wr00721 FIG. 52 illustrates an example of an asymmetric imaging relay 
spectrograph in accordance with an embodiment of the present invention using simulated 
DMD tilted object, two paraxial lenses, a grating and a tilted FPA at the image plane; 

fQQ?3^r0073] F IG. 53 illustrates an imaging lens and TIR prism example for 
improving F numbers of operation using the DMD as an SLM in accordance with an 
embodiment of the present invention; 

|"0073i r0074] FIG. 54 illustrates a schematic diagram of the hyper-spectral 
imaging system in accordance embodiment of the present invention; 
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fQQ74^fQ0751 FIG, 55 is an exemplary hyper-spectral imaging system in 
accordance with an embodiment of the present invention; 

fOQ?S^[00761 F IG. 56 depicts an optimized model of an all reflective image 
delivery system for a DMD enabled SLM spectral imaging system or imaging system in 
accordance with an embodiment of the present invention; 

fQQ764r00771 FIG. 57 illustrates an optical relay system in accordance with an 
embodiment of the present invention optimized for use with a TIR prism assembly and 
the DMD SLM; 

ffiQ??4r00781 FIG. 58 illustrates a CAD drawing of NSTIS in accordance with an 
embodiment of the present invention; 

fOO7^[0079] FIG. 59 is an example of the mapping of micro-mirror columns to 
the dispersed spectral images at the focal plane of a 2D array sensor, FPA or camera in 
accordance with an embodiment of the present invention: and 

fQQ794r00801 FIG. 60 illustrates a conventional pushbroom spectral imaging 
spectral system^t 

[00 8 0JFIG. 61 illustrat e s a conv e ntional liquid crystal tunabl e filt e r scann e d spectral 
imaging system; and 

[0081] FIG. 62 illustrat e s a conv e ntional scanning multipl e xed spectral imaging syst e m, 
such as a Fouri e r transform focal plan e array spectral imaging syst e m. 

DETAILED DESCRIPTION OF THE EMBODIMENTS 

fOQ83^r0081] Tuminp now to the drawing figures and particularly Fig. lA and 
IB, a spectrometer assembly 10 constructed in accordance with one embodiment of the 
invention is illustrated. With reference to Fig. 1 A the device broadly includes a source 12 
of electromagnetic radiation, a mirror and slit assembly 14, a wavelength dispersing 
device 16, a spatial light modulator 18, a detector 20, and an analyzing device 22. 

fQ0&^r00821 In particular, the electromagnetic radiation source 12 is operable to 
project rays of radiation onto or through a sample 24 that is to be analyzed, such as a 
sample of body tissue or blood. The radiation source can be any device that generates 
electromagnetic radiation in a known wavelength spectrum such as a globar, hot wire, or 
light bulb that produces radiation in the infi*ared spectrum. To increase the amount of 
rays that are directed to the sample, a parabolic reflector 26 can be interposed between the 
source 12 and the sample 24. In a specific embodiment, the source of electromagnetic 
radiation is selected as to jdeld a continuous band of spectral energies, and is referred to 
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as the source radiation. It should be apparent that the energies of the radiation source are 
selected to cover the spectral region of interest for the particular application. 

ffiQ844r00831 The mirror and slit assembly 14 is positioned to receive the 
radiation rays from the source 12 after they have passed through the sample 24 and is 
operable to focus the radiation onto and through an entrance slit 30. The collection mirror 
28 focuses the radiation rays through slit 30 and illuminates the wavelength dispersing 
device 16. As shown in diagram form in Fig. IB, in different embodiments of the 
invention radiation rays from the slit can also be collected through a lens 15, before 
illuminating a wavelength dispersion device 16. 

r00851 [00841 The wavelength dispersing device 16 receives the beams of 
radiation from the mirror and slit assembly 14 and disperses the radiation into a series of 
lines of radiation each corresponding to a particular wavelength of the radiation spectrum. 
The preferred wavelength dispersing device is a concave diffraction grating; however, 
other wavelength dispersing devices, such as a prism, can be utilized. In a specific 
embodiment, the wavelengths from the dispersing device 16 are in the near infrared 
portion of the spectrum and can cover, for example, the range of 1650-1850 nanometers 
(nm). It should be emphasized, however, that in general this device is not limited to just 
this or to any spectral region. It is intended that the dispersion device in general is 
capable of operating in other ranges of electromagnetic radiation, including ' the 
ultraviolet, visible, infrared, and microwave spectrum portions, as well as acoustic, 
electric, magnetic, and other signals, where applicable. 

fQ086^[ 0085] The spatial light modulator (SLM) 18 receives radiation from the 
wavelength dispersing device 16, individually modulates each spectral line, and reflects 
the modulated lines of radiation onto the detector 20. As illustrated in Fig. 2, the SLM is 
implemented in a first preferred embodiment as a micro-mirror array that includes a semi- 
conductor chip or piezo-electric device 32 having an array of small reflecting surfaces 34 
thereon that act as minors. One such micro-mirror array is manufactured by Texas 
Instruments and is described in more detail in U.S. Pat. No. 5,061,049, hereby 
incorporated into the present application by reference. Those skilled in the art will 
appreciate that other spatial light modulators, such as a magneto-optic modulator or a 
liquid crystal device can be used instead of the micro-mirror array. Various embodiments 
of such devices are discussed in more detail below. 

fOQ&^[ "0Q86] The semi-conductor 32 of the micro-mirror array 18 is operable to 
individually tilt each mirror along its diagonal between a first position depicted by the 
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letter A and a second position depicted by the letter B in Fig. 3. In preferred forms, the 
semi-conductor tilts each mirror 10 degrees in each direction from the horizontal. The 
tilting of the mirrors 34 is preferably controlled by the analyzing device 22, which can 
communicate with the micro-mirror array 1 8 through an interface 37. 

fQO8&4r00871 The micro-mirror array 18 is positioned so that the wavelength 
dispersing device 16 reflects each of the lines of radiation upon a separate colunm or row 
of the array. Each column or row of mirrors is then tilted or wobbled at a specific and 
separate modulation frequency. For example, the first row of mirrors can be wobbled at a 
modulation frequency of 100 Hz, the second row at 200 Hz, the third row at 300 Hz, etc. 

[0089i r00881 In a specific embodiment, the mirrors are calibrated and positioned 
so that they reflect all of the modulated lines of radiation onto a detector 20, Thus, even 
though each colunm or row of mirrors modulates its corresponding line of radiation at a 
different modulation frequency, all of the lines of radiation are focused onto a single 
detector. 

fQQ9Q^r00891 T he detector 20, which can be any conventional radiation 
transducer or similar device, is oriented to receive the combined modulated lines of 
radiation from the micro-mirror array 18. The detector is operable for converting the 
radiation signals into a digital output signal that is representative of the combined 
radiation lines that are reflected from the micro-mirror array. A reflector 36 can be 
interposed between the micro-mirror array 1 8 and the detector 20 to receive the combined 
modulated lines of radiation from the array and to focus the reflected lines onto the 
detector. 

fOQ944r00901 T he analyzing device 22 is operably coupled with the detector 20 
and is operable to receive and analyze the digital output signal from the detector. The 
analyzing device uses digital processing techniques to demodulate the signal into separate 
signals each representative of a separate line of radiation reflected from the micro-mirror 
array. For example, the analyzing device can use discrete Fourier transform processing to 
demodulate the signal to determine, in real time, the intensity of each line of radiation 
reflected onto the detector. Thus, even though all of the lines of radiation from the micro- 
mirror array are focused onto a single detector, the analyzing device can separately 
analyze the characteristics of each line of radiation for use in analyzing the composition 
of the sample. 

fOO934r0091] In accordance with one embodiment of this invention, the 
analyzing device is preferably a computer that includes spectral analysis software. Fig. 4 
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illustrates an output signal generated by the analyzing device in accordance with one 
embodiment. The output signal illustrated in Fig. 4 is a plot of the absorption 
characteristics of five wavelengths of radiation fi-om a radiation source that has passed 
through a sample. 

fOQ9^r00921 In one embodiment of the system of this invention illustrated in 
Fig. 6A, it is used for digital imaging purposes. In particular, when used as an imaging 
device, an image of a sample 38 is focused onto a micro-mirror array 40 and each micro- 
mirror in the array is modulated at a different modulation rate. The micro-mirror array 
geometry is such that some or all of the reflected radiation impinges upon a single 
detector element 42 and is subsequently demodulated to reconstruct the original image 
improving the signal-to-noise ratio of the imager. Specifically, an analyzing device 44 
digitally processes the combined signal to analyze the magnitude of each individual pixel. 
Fig. 6B illustrates spatio-spectral distribution of the DMA, where individual elements can 
be modulated. Fig. 5 is a plot of a three dimensional image showing the magnitude of 
each individual pixel. 

fO0944|'0093] Fig. 7 illustrates the output of a digital micro-mirror array (DMA) 
filter spectrometer used as a variable band pass filter spectrometer, variable band reject 
filter spectrometer, variable multiple band pass filter spectrometer or variable multiple 
band reject filter spectrometer. In this embodiment, the combined measurement of the 
electromagnetic energy absorbed by sample constituents A and C is of interest. The 
shaded regions in Fig. 7 illustrate the different regions of the electromagnetic spectrum 
that will be allowed to pass to the detector by the DMA filter spectrometer. The 
wavelengths of electromagnetic radiation selected to pass to the detector correspond to 
the absorption band for compound A and absorption band for compound C in a sample 
consisting of compounds A, B, and C. The spectral region corresponding to the 
absorption band of compound B and all other wavelengths of electromagnetic radiation 
are rejected. Those skilled in the art will appreciate that the DMA filter spectrometer is 
not limited to the above example and can be used to pass or reject any combination of 
spectral resolution elements available to the DMA. Various examples and modifications 
are considered in detail below. 

r0095i r0094] As a DMA filter imager the spatial resolution elements (pixels) of 
an image can be selectively passed or rejected (filtered) according to the requirements of 
the image measurement. The advantages of both the DMA filter spectrometer and DMA 
filter imager are: 
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(1) All spectral resolution elements or spatial resolution elements 
corresponding to the compounds of interest in a particular sample can be directed 
simultaneously to the detector for measurement. This has the effect of increasing the 
signal-to-noise ratio of the measurement. 

(2) The amount of data requiring processing is reduced. This reduces storage 
requirements and processing times. 

ffi6964r00951 As noted above, using a DMA one can provide one or more 
spectral band pass or band-reject filter(s) with a chosen relative intensity. In particular, in 
accordance with the present invention the radiation wavelengths that are reflected in the 
direction of the detector are selected by specific columns of micro-mirrors of the DMA, 
as illustrated in Fig. 8. The relative intensity of the above spectral band is controlled by 
the selection of specific area of micro-mirrors on the DMA, represented by the dark area 
designated "A" in Fig. 8. Thus, the dark area shown in Fig. 8 is the mirrors that direct 
specific wavelength radiation, i.e., spectral band, to the detector. Clearly, the "on" minors 
in the dark area create a band-pass filter, the characteristics of which are determined by 
the position of the "on" area in the DMA. The bottom portion of the figure illustrates the 
profile of the radiation reaching the detector. 

r0097i r00961 F ig. 8 also demonstrates the selection of specific rows and columns 
of mirrors in the DMA used to create one spectral band filter with a single spectral mode. 
It should be apparent, however, that using the same technique of blocking areas in the 
DMA one can obtain a plurality of different specific spectral band filters, which can have 
multi-modal characteristics. The design of such filters is illustrated in Fig. 9. 

fOO9&4r00971 A s shown in Fig. 9, a multitude of different specific filters can be 
designed on one DMA using simple stacking. Fig. 9 illustrates the creation of several 
filters by selective reflection from specific micro-mirrors. In particular, the left side of 
the figure illustrates the creation of three different filters, designated 1, 2, and 3. This is 
accomplished by the selection of specific mirrors on the DMA, as described above with 
reference to Fig. 8. The total collection of spectral band filters is shown at the bottom-left 
of this figure. The spectral band provided by each filter is shown on the right-hand side 
of the figure. The bottom right portion illustrates the radiation passing through the 
combination of filters 1, 2 and 3. 

fQ0994r0098] The above discussion describes how the relative intensity of each 
spectral band can be a function of the DMA area used in the reflection. The following 
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table illustrates the linear relationship between areas of the DMA occupied by individual 
filters, and the resulting filter. Clearly, if the entire DMA array is in the "on" position, 
there will be no filtering and in principle the input radiation passes through with no 
attenuation. 



Figure 9, left side 


Figure 9, right side 


Reflected radiation fi-om micro-mirrors 


Filter created 


area A 


1 


area B 


2 


area C 


3 


areas a + b + c 


1 +2 + 3 







rOOl Q0i r00991 Figure 10 illustrates the means for the intensity variation of a 
spectral filter built in accordance with this invention, and is summarized in the table 



below. 



Example A 

Reflection fi"om a DMA 
See Figs. 8 and 9. 

Reflection areas 1, 2, and 3 create spectral 
filter 1 , 2 and 3 respectively, 
area 1 = area 2 = area 3 


Example B 

The intensity recorded at the detector for 
example A for the combination filter 1, 2, 
and 3, Intensity, I, Ii = I2 = I3 


Example C 

The reflection of area 2 of the DMA is 

increased. 

area 1 = area 3 < area 2 


Example D 

The intensity recorded at the detector for 
filters 1, 2, and 3 is 

Ii . KI2 


Example E 

The reflection of area 2 of the DMA is 
decreased 

area 1= area 3 > area 2 


Example F 

The intensity recorded at the detector for 
filter 1, 2, and 3 is 

II ^I3>l2 



fOO4 ^r001001 Figures 9 and 10 illustrate the ability to design spectral 

filters with different characteristics using a DMA. A point to keep in mind is that 
different spectral components of the radiation from the sample have been separated in 
space and can be filtered individually. The ability to process individual spectral 
components separately should be retained. To this end, in accordance with the present 
invention, spectral components are modulated. 
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rQ0l02i r00l0l1 The basic idea is to simply modulate the output from 
different filters differently, so one can identify and process them separately. In a 
preferred embodiment, different modulation is implemented by means of different 
modulation rates. Thus, with reference to Fig. 9, the output of filter 1 is modulated at rate 
Mi; output of filter 2 is modulated at rate M2, and filter 3 is modulated using rate M3, 
where M| ^ M3. In different embodiments, modulation can be achieved by assigning 
a different modulation encodement to each filter, with which it is modulated over time. 

r00103i rQQ102] As a result, a system built in accordance with the present 

invention is capable of providing: a) Spectral bandwidth by selection of specific columns 
of micro-mirrors in an array; b) Spectral intensity by selection of rows of the array; and c) 
Spectral band identification by modulation. 

rOO 1 0 4 1 rOO 1 03 1 F i gures 11-14 illustrate alternative embodiments of a 
modulating spectrometer in accordance with this invention, where the DMA is replaced 
with different components. In particular. Fig. 11 A and B show an embodiment in which 
the DMA is replaced with fixed elements, in this case concave mirrors. The idea is to use 
fixed spectral grating, which masks out spectrum block components that are not needed 
and passes those which are. 

[Q0105i r0 0104] The idea here is that the broadly illuminated dispersive 

element distributes spectral resolution elements in one dimension so that in the orthogonal 
dimension one can collect light of the same wavelengths. With reference to Fig. 6A one 
can see that at a particular plane, herein called the focal plane, one has a wavelength 
axis(x or columns) and a spatial axis(y or rows). If one were to increase the number of 
spatial resolution elements (y) that are allowed to pass energy through the system and out 
of the exit aperture for any given wavelength (x), or spectral resolution element (x), this 
would have the effect of increasing the intensity of the particular spectral resolution 
elements' intensity at the detector. 

rOOl 061 [001051 If the array of spatio/spectral resolution elements at the 
focal plane as shown in Fig. 6A is replaced with fixed elements, such as the concave 
mirrors in Fig. IIB, one can have a different device configured to perform a particular 
signal processing task - in this case pass the predetermined spectrum components at the 
desired intensity levels. Fig. IIA shows the spatio/spectral resolution elements at the 
focal plane to be used. The fixed optical elements are placed to interact with 
predetermined spatio/spectral resolution elements provided by the grating and entrance 
aperture geometry and to direct the specific assortment of spatio/spectral elements to 
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specific spatial locations for modulation encoding (possibly using the barber pole 
arrangement, shown next). 

rOO 1 071 rOO 1 061 Fi p. 12 illustrates an embodiment of a complete modulating 
spectrometer in which the DMA element is replaced by the concave mirrors of Fig. 1 1 . 
Figure 13 illustrates a modulating lens spectrometer using lenses instead of DMA, and a 
"barber pole" arrangement of mirrors to implement variable modulation. The "barber 
pole" modulation arrangement is illustrated in Fig. 14. 

[00 1 081 [00 1 07] With reference to Fig. 14, modulation is accomplished by 

rotating this "barber pole" that has different number of mirrors mounted for reflecting 
light from the spatially separated spectral wavelengths. Thus, irradiating each vertical 
section will give the reflector its own distinguishable frequency. In accordance with this 
embodiment, light from the pole is collected and simultaneously sent to the detector. 
Thus, radiation from concave mirror 1 impinges upon the four-mirror modulator; concave 
mirror 2 radiation is modulated by the five-mirror modulator, and concave mirror 3 
directs radiation to the six-mirror modulator. In the illustrated embodiment, the 
modulator rate is four, five, or six times per revolution of the "barber pole." 

[00 1091 [00 1081 The operation of the device is clarified with reference to 
Fig. 12, tracing the radiation from the concave mirrors 12 to the detector of the system. 
In particular, concave mirror 1 reflects a selected spectral band with chosen intensity. 
This radiated wave impinges upon a modulator, implemented in this embodiment as a 
rotation barber pole. The modulating rates created by the barber pole in the exemplary 
embodiment shown in the figure are as shown in the table below. 





Number of mirrors 


Modulation 


Figure 13 


Per 360° rotation 


Per 360° of barber pole 


Area A 


4 


4/360° 


Area B 


5 


5/360° 


Area C 


6 


6/360° 



[001 101 1^00109] A ccordinglv- this arrangement yields a modulation rate of 
4/360° for the radiation from Area A, Figure 12. 

[OOllli rOOllOl By a analogy, the mirrors of Areas B and C are modulated 

at the rate of 5/360° and 6/360°, respectively. As illustrated, all radiation from mirrors A, 
B, and C is simultaneously directed to the detector. This radiation is collected by either a 
simple mirror lens or a toroidal mirror, which focuses the radiation onto a single detector. 
The signal from the detector now goes to electronic processing and mathematical analyses 
for spectroscopic results. 
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[QQl 12i rQQl 111 In the discussion of modulating spectrometers, a single light 
source of electromagnetic radiation was described. There exist yet another possibility for 
a unique optical design - a modulating multi-light source spectrometer. Figs. 15 and 16 
illustrate an embodiment of this invention in which a light source 12 provides several 
modulated spectral bands, e.g., light emitting diodes (LED), or lasers (shown here in three 
different light sources). The radiation from these light sources impinges upon the sample 
24. One possible illumination design is one in which light from a source, e.g. LED, 
passes through a multitude of filters, impinging upon the sample 24. The radiation from 
the sample is transmitted to a detector 20, illustrated as a black fiber. The signal from the 
detector is electronically processed to a quantitative and qualitative signal describing the 
sample chemical composition. 

r00113i r001121 in this embodiment, a plurality of light sources is used at 
differed modulating rates. Fig. 15 and 16 illustrate the combination of several light 
sources in the spectrometer. The choice of several different spectral bands of 
electromagnetic radiation can be either light emitting diodes, LED, lasers, black body 
radiation and/or microwaves. Essentially the following modulation scheme can be used 
to identify the different light sources, in this example LED's of different spectral band 
wavelength. 



No. of Spectral band Modulation 

Source Wavelength, rim Rate 

1 1500-1700 m, 

2 1600-1800 m2 

3 1700-1900 ms 



Note: mi:^m29^m39^:... 

fQOj44;r00ll31 It should be noted that either the radiation will be scattered 
or transmitted by the sample 24. This scattered or transmitted radiation from the sample 
is collected by an optical fiber. This radiation from the sample is conducted to the 
detector. The signal from the detector is electronically processed to yield quantitative and 
qualitative information about the sample. 

rOOl ISi rOOl 14] in a particular embodiment the radiation path consists of 
optical fibers. However, in accordance with alternate embodiments, mirrors and lenses 
could also constitute the optical path for a similar modulating multi-light source 
spectrometer. 
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r00116i rQ01151 The spectrometer described herein records spectral 
information about one unique area on a single detector. In a similar manner, the spectral 
characteristic of a multitude of areas in a sample can be recorded with a multitude of 
detectors in accordance with different embodiments of the invention. Such a multitude of 
detectors exists in an array detector. Array detectors arc known in the art and include, for 
example 

fOQ44^[Q0 1 1 61 Charge coupled devices (CCD), in the ultraviolet, and 

visible portions of the spectrum; InSb - array in near infrared; hiGaAs - array in near 
infrared; Hg-Cd-Te - array in mid-infrared and other array detectors. 

fOOl 1 81 TOO 1 1 71 Array detectors can operate in the focal plane of the optics. 
Here each detector of the array detects and records the signal from a specific area, xiy. 
Practical Example B described herein on the gray-level camera provides a further 
illustration. Different aspects of the embodiments discussed herein are considered in 
more detail. As is understood by one skilled in the art, standard optical duality implies 
that each of the preceding configurations can be operated in reverse, exchanging the 
position of the source and the detector. 

r00119i rQ01181 T he postsample processing, i.e., signal processing 
performed after a sample had been irradiated, describes an aspect of the present invention. 
In accordeince with another aspect of this invention, significant benefits can result from 
irradiating a sample with pre-processed radiation, in what is referred to as pre-sample 
processing. In accordance with an embodiment of the present invention, one or more 
light sources, capable of providing modulated temporal and/or spatial patterns of input 
radiation, should be used. These sources are referred to next as controllable source(s) of 
radiation, which in general are capable of generating arbitrary combinations of spectral 
radiation components within a predetermined spectrum range. 

["001 20] [001 191 Several types of prior art devices are known that are 
capable of providing controllable radiation. Earlier prior art devices primarily relied upon 
various "meisking" techniques, such as electronically alterable masks interposed in the 
optical pathway between a light source and a detector. More recent prior art devices use a 
combination of two or more light-emitting diodes (LEDs) as radiation sources. Examples 
are provided in U.S. Pat. Nos. 5,257,086 and 5,488,474, the content of which is hereby 
incorporated by reference for all purposes. As discussed in the above patents, an array of 
LEDs or light-emitting lasers is configured for activation using a particular encoding 
pattern, and can be used as a controllable light source. A disadvantage of this system is 
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that it relies on an array of different LED elements, each operating in a different, 
relatively narrow spectrum band. In addition, there are technological problems associated 
with having an array of discrete radiation elements with different characteristics. 

rOO 1211 TOO 1 201 T hese and other problems associated with the prior art are 
addressed in accordance with the present invention using a device that in a specific 
embodiment can be thought of as the reverse of the setup illustrated in Fig. lA. In 
particular, one or more broadband radiation sources illuminate the digital micro-mirror 
array (DMA) 1 8 and the modulations of the micro-mirrors in the DMA encode the source 
radiation prior to impinging upon the sample. The reflected radiation is then collected 
from the sample and directed onto a detector for further processing. 

rQQ122i r001211 Fig. 17 illustrates a schematic representation of an 

apparatus in accordance with the present invention using a controllable radiation source. 
Generally, the system includes a broadband radiation source 12, DMA 18, wavelength 
dispersion device 16, slit assembly 30, detector 20 and control assembly 22. 

r00123i r001221 In particular, control assembly 22 can include a 
conventional personal computer 104, interface 106, pattern generator 108, DMA driver 
110, and analog to digital (A/D) converter 114. Interface 106 operates £is a protocol 
converter enabling communications between the computer 22 and devices 108-1 14. 

fflOt344r00 1 231 Pattem generator 108 can include an EPROM memory 
device (not shown) which stores the various encoding patterns for array 18, such as the 
Hadamard encoding pattem discussed below. In response to control signals from 
computer 22, generator 108 delivers signals representative of successive pattems to driver 
110. More particularly, generator 108 produces output signals to driver 110 indicating 
the activation pattem of the mirrors in the DMA 18. A/D converter 1 14 is conventional 
in nature and receives the voltage signals from detector 20, amplifies these signals as 
analog input to the converter in order to produce a digital output representative of the 
voltage signals. 

fOQl^rOOl 241 R adiation source 12, grating 16, DMA 18 sUt assembly 30 
and detector 20 cooperatively define an optical pathway. Radiation from source 12 is 
passed through a wavelength dispersion device, which separates in space different 
spectrum bands. The desired radiation spectrum can them be shaped by DMA 1 8 using 
the filter arrangement outlined herein. In accordance with a preferred embodiment, 
radiation falling on a particular micro-mirror element can also be encoded with a 
modulation pattem applied to it. In a specific mode of operating the device, DMA 18 is 
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activated to reflect radiation in a successive set of encoding patterns, such as Hadamard, 
Fourier, wavelet or others. The resultant set of spectral components is detected by 
detector 20, which provides corresponding output signals. Computer 22 then processes 
these signals. 

rOO 1 261 TOO 1251 Computer 22 initiates an analysis by prompting pattem 
generator 108 to activate the successive encoding pattems. With each pattem, a set of 
wavelength components are resolved by grating 1 6 and after reflection from the DMA 1 8 
is directed onto detector 20. Along with the activation of encoding pattems, computer 22 
also takes readings from A/D converter 114, by sampling data. These readings enable 
computer 22 to solve a conventional inverse transform, and thereby eliminate background 
noise from the readings for analysis. 

r00127i r001261 In summary, the active light source in accordance with the 
present invention consists of one or more light sources, from which various spectral bands 
are selected for transmission, while being modulated with a temporal £ind/or spatial 
pattems. The resulting radiation is then directed at a region (or material) of interest to 
achieve a variety of desired tasks. A brief listing of these tasks include: (a) Very precise 
spectral coloring of a scene, for purposes of enhancement of display and photography; (b) 
Precise illumination spectrum to correspond to specific absorption lines of a compound 
that needs to be detected, (see figures 40-44 on protein in wheat as an illustration) or for 
which it is desirable to have energy absorption and heating, without affecting neighboring 
compounds (This is the principle of the microwave oven for which the radiation is tuned 
to be absorbed by water molecules allowing for heating of moist food only); (c) The 
procedure in (b) could be used to imprint a specific spectral tag on ink or paint, for 
watermarking, tracking and forgery prevention, acting as a spectral bar code encryption; 
(d) The process of light curing to achieve selected chemical reactions is enabled by the 
tunable light source. 

rOO 1 281 rOO 1 271 various other applications are considered herein. Duality 
allows one to reverse or "tum inside out" any of the post-sample processing 
configurations described previously, to yield a pre-sample processing configuration. 
Essentially, in the former case one takes post sample light, separates wavelengths, 
encodes or modulates each, and detects the result. The dualized version for the latter case 
is to take source light, separates wavelengths, encode or modulate each, interact with a 
sample, and detect the result. 



60203834,1 



26 



PSS 212.2 US 



rOO 1 201 rOQ 1 281 Various embodiments of systems for perfomiing post- and 
pre-sample processing were discussed herein. In a specific embodiment, the central 
component of the system is a digital micro-mirror array (DMA), in which individual 
elements (micro-mirrors) can be controlled separately to either pass along or reject certain 
radiation components. By the use of appropriately selected modulation pattems, the 
DMA array can perform various signal processing tasks. In a accordance with a preferred 
embodiment of this invention, the functionality of the DMAs discussed above can be 
generalized using the concept of Spatial Light Modulators (SLMs), devices that broadly 
perform spatio-spectral encoding of individual radiation components, and of optical 
synapse processing units (OSPUs), basic processing blocks. This generalization is 
considered herein as well as the Hadamard processing, spatio-spectral tagging, data 
compression, feature extraction and other signal processing tasks. 

r00130i r001291 i n accordance with the present invention, one-dimensional 
(ID), two-dimensional (2D) or three-dimensional (3D) devices capable of acting as a 
light valve or array of light valves are referred to as spatial light modulators (SLMs). 
More broadly, an SLM in accordance with this invention is any device capable of 
controlling the magnitude, power, intensity or phase of radiation or which is otherwise 
capable of changing the direction of propagation of such radiation. This radiation can 
either have passed through, or be reflected or refi-acted from a material sample of interest. 
In a preferred embodiment, an SLM is an array of elements, each one capable of 
controlling radiation impinging upon it. Note that in accordance with this definition an 
SLM placed in appropriate position along the radiation path can control either spatial or 
spectral components of the impinging radiation, or both. Furthermore, "lighf is used 
here in a broad sense to encompass any portion of the electromagnetic spectrum and not 
just the visible spectrum. Examples of SLM's in accordance with different embodiments 
of the invention include liquid crystal devices, actuated micro-mirrors, actuated mirror 
membranes, di-electric light modulators, switchable filters and optical routing devices, as 
used by the optical communication and computing environments and optical switches. In 
a specific embodiment, the use of a DMA as an example of spatial light modulating 
element is discussed herein. U.S. Pat. No. 5,037,173 provides examples of technology 
that can be used to implement SLM in accordance with this invention, and is hereby 
incorporated by reference. 

fOm^rOO nOl in a preferred embodiment, a ID, 2D, or 3D SLM is 

configured to receive any set of radiation components and fixnctions to selectively pass 
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these components to any number of receivers or image planes or collection optics, as the 
application can require, or to reject, reflect or absorb any input radiation component, so 
that either it is or is not received by one or more receivers, image planes or collection 
optics devices. It should be clear that while in the example discussed herein, the SLM is 
implemented as a DMA, virtually any array of switched elements can be used in 
accordance with the present invention. 

rOOl 32i r001 3 1 1 Generallv. an SLM in accordance with the invention is 
capable of receiving any number of radiation components, which are then encoded, 
tagged, identified, modulated or otherwise changed in terms of direction and/or 
magnitude to provide a unique encodement, tag, identifier or modulation sequence for 
each radiation component in the set of radiation components, so that subsequent optical 
receiver(s) or measuring device(s) have the ability to uniquely identify each of the input 
radiation components and its properties. In a relevant context, such properties include, 
but are not limited to, irradiance, wavelength, band of fi-equencies, intensity, power, phase 
and/or polarization. The tagging of individual radiation components can be accomplished 
using rate modulation. Thus, different spectral components of the input radiation that 
have been separated in space using a wavelength dispersion device are then individually 
encoded by modulating the micro-mirrors of the DMA array at different rates. The 
encoded radiation components are directed to a single detector, but nevertheless can be 
analyzed individually using Fourier analysis of the signal fi-om the detector. Other 
examples for the use of "tagging" are discussed below. 

r00133i r001321 In accordance with this invention, various processing 

modalities can be realized with an array of digitally controlled switches (an optical 
synapse), which function to process and transmit signals between different components of 
the system. In the context of the above description, the basic OSPU can be thought of as 
a data acquisition unit capable of scanning an array of data, such as an image, in various 
modes, including raster, Hadamard, multiscale wavelets, and others, and transmitting the 
scanned data for fiirther processing. Thus, a synapse is a digitally controlled array of 
switches used to redirect image (or generally data) components or combinations of light 
streams, from one location to one or more other locations. In particular it can perform 
Hadamard processing, as defined below, on a plurality of radiation elements by 
combining subsets of the elements (i.e., binning) before conversion to digital data. A 
synapse can be used to modulate light streams by modulating temporally the switches to 
impose a temporal bar code (by varying in time the binning operation). This can be built 
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in a preferred embodiment from a DMA, or any of a number of optical switching or 
routing components, used for example in optical communications applications. 

r0013 4 i r001331 An OSPU unit in accordance with the present invention is 
shown in diagram form in Fig. 18A and 18B, as three-port device taking input from a 
radiation source S, and distributing it along any of two other paths, designated C (short 
for camera) and D (for detector). Different scanning modes of the OSPU are considered 
in more detail herein. 

r00l351 [00l341 In the above disclosure and in one preferred embodiment of 

the invention an OSPU is implemented using a DMA, where individual elements of the 
array are controlled digitally to achieve a variety of processing tasks while collecting 
data. In accordance with the present invention, information bearing radiation sources 
could be, for example, a stream of photons, a photonic wavefront, a sound wave signal, an 
electrical signal, a signal propagating via an electric field or a magnetic field, a stream of 
particles, or a digital signal. Example of devices that can act as a synapse include spatial 
light modulators, such as LCDs, MEMS mirror arrays, or MEMS shutter arrays; optical 
switches; optical add-drop multiplexers; optical routers; and similar devices configured to 
modulate, switch or route signals. Clearly, DMAs and other optical routing devices, as 
used by the optical communication industry can be used to this end. It should be apparent 
that liquid crystal displays (LCD), charge coupled devices (CCD), CMOS logic, arrays of 
microphones, acoustic transducers, or antenna elements for electromagnetic radiation and 
other elements with similar functionality that will be developed in the fiiture, can also be 
driven by similar Methods. 

rOQl 3 61 TOO 1 351 A pplicants' contribution in this regard is in the novel 

process of performing pretransduction digital computing on analog data via adaptive 
binning means. Such novelty can be performed in a large number of ways. For example, 
one can implement adaptive current addition using a parallel/serial switch and wire 
networks in CMOS circuits. Further, in the acoustic processing domain, one or more 
microphones can be used in combination with an array of adjustable tilting sound 
reflectors (like a DMD for soimd). In each case, one can "bin" data prior to transduction, 
in an adaptive way, and hence measure some desired computational result that would 
traditionally be obtained by gathering a "data cube" of data, and subsequently digitally 
processing the data. The shift of paradigm is clear: in the prior art traditionally analog 
signals are captured by a sensor, digitized, stored in a computer as a "data cube", and then 
processed. Considerable storage space and computational requirements are extended to 
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do this processing. In accordance with the present invention, data from one or more 
sensors is processed directly in the analogue domain, the processed result is digitized and 
sent to a computer, where the desired processing result can be available directly, or 
following reduced set of processing operations. 

r00137i r001361 I n accordance with the present invention, the digitally 
controlled array is used as a hybrid computer, which through the digital control of the 
array elements performs (analog) computation of inner products or more generally of 
various correlations between data points reaching the elements of the array and prescribed 
pattems. The digital control at a given point (i.e., element) of the array can be achieved 
through a variety of different mechanisms, such as applying voltage differences between 
the row and column intersecting at the element; the modulation is achieved by addressing 
each row and column of the array by an appropriately modulated voltage pattern. For 
example, when using DMA, the mirrors are fluctuating between two tilted positions, and 
modulation is achieved through the mirror controls, as known in the art. The specifics of 
providing to the array element of signal(s) following a predetermined pattem will depend 
on the design implementation of the array and are not considered in further detail. 
Broadly, the OSPU array is processing raw data to extract desired information. 

r00l38i r00l371 In accordance with the present invention, various 

assemblies of OSPU along with other components can be used to generalize the ideas 
presented above and enable new processing modalities. For example, Fig. 19 illustrates 
in block diagram form the design of a spectrograph using OSPU. As shown, the basic 
design brings reflected or transmitted radiation from a line in the sample or source onto a 
dispersing device 1 6, such as a grating or prism, onto the imaging fiber into the OSPU to 
encode and then forward to a detector 20. 

r00139i r001381 Fig. 20 illustrates in a diagram form an embodiment of a 
tunable light source, which operates as the spectrograph in Fig. 19, but uses a broadband 
source. In this case, the switching elements of the OSPU array, for example the mirrors 
in a DMA, are set to provide a specified energy in each row of the mirror, which is sent to 
one of the outgoing imaging fiber bundles. This device can also fiinction as a 
spectrograph through the other end, i.e., fiber bundle providing illumination, as well as 
spectroscopy. 

r001 4 0i r00139] Fie. 21 illustrates in a diagram form an embodiment of the 

spectral imaging device discussed herein, which is built with two OSPUs. Different 
configurations of generalized processing devices are illustrated in Fig. 22, in which each 
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side is imaging in a different spectral band, and Fig. 23, which illustrates the main 
components of a system for processing input radiation using an OSPU. 

[001in r001401 in accordance with the present invention, different scanning 
modes can be used in different applications, as illustrated in Fig. 24, Fig. 25 and Fig. 26. 
These algorithms are of use, for example, when one is using an OSPU in conjunction with 
a single sensor, and the OSPU is binning energy into that sensor, the binning being 
determined by the pattern that is put onto the SLM of the OSPU. 

[00112i r 001411 In particular. Fig. 24 is a flow chart of a raster-scan using in 

one embodiment of the present invention. This algorithm scans a rectangle, the "Region 
Of Interest (ROI)," using ordinary raster scanning. It is intended for use in configurations 
in this disclosure that involve a spatial light modulator (SLM). It is written for the 2D 
case, but the obvious modifications will extend the algorithm to other dimensions, or 
restrict to ID. 

fflQM^r001421 F ig. 25 is a flowchart of a Walsh-Hadamard scan used in 
accordance with another embodiment of the invention. This algorithm scans a rectangle, 
the "Region Of Interest (ROI)", using Walsh-Hadamard multiplexing. Walsh( dx, m, i, 
dy, n, j) is the Walsh-Hadamard pattem with origin (dx, dy), of width 2™ and height 2", 
horizontal Walsh index i, and vertical Walsh index j. 

r00111i r0 0143] Fig. 26 is a flowchart of a multi-scale scan. This algorithm 

scans a rectangle, the "Region Of Interest (ROI)", using a multi-scale search. It is 
intended for use in a setting as in the description of the raster scanning algorithm. The 
algorithm also presumes that a procedure exists for assigning a numerical measure to the 
pattem that is currently on is called an "interest factor." 

rQ01 4 5'| [00144] F ig. 26A illustrates a multi-scale tracking algorithm in a 
preferred embodiment of the present invention. The algorithm scans the region of 
interest, (using multi-scan search), to find an object of interest and then tracks the object's 
movement across the scene. It is intended for use in a setting where multi-scale search 
can be used, and where the "interest factor" is such that a trackable object can be found. 
Examples of interest factors used in accordance with a preferred embodiment (when 
pattem L, is put onto the SLM, the sensor reads Ci and we are defining the "interest 
factor" Fi) in the preceding scan algorithms a single sensor is assumed. Thus 

1. F(Li) = Q 

2. F(Li) = Ci / area(Li) 
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3 F(Li) = Ci / Ck, where Lk is the rectangle that contains Lj, and that has N 
times the area of Lj, (for example, N=4), and which has already been scanned by the 
algorithm (there will always be exactly one such). 

[QOHdl fOOMSI A modification of the algorithm is possible, where instead 
of putting up the pattern Lj, one can put up a set of a few highly oscillatory Walsh 
pattems fully supported on exactly Li, and take the mean value of the sensor reading as Fj. 
This estimates the total variation within Li and will yield an algorithm that finds the edges 
within a scene. In different examples the sensor is a spectrometer. F(Li) = distance 
between the spectrum read by the sensor, and the spectrum of a compound of interest 
(distance could be, e.g., Euclidean distance of some other standard distance). This will 
cause the algorithm to zoom in on a substance of interest. 

r00147i rQ01461 In another embodiment, F(Li) = distance between the 
spectrum read by the sensor, and the spectrum already read for Lk, where Lk is the 
rectangle that contains Li, and that has N (N=4) times the area of Li, and which has 
already been scanned by the algorithm (there will always be exactly one such). This will 
cause the algorithm to zoom in on edges between distinct substances. 

rOO 1 4 81 TOO 1471 In yet another embodiment, F(Li) = distance between the 
spectmm read by the sensor, and the spectrum already read for Lq. This will cause the 
algorithm to zoom in on substances that are anomalous compared to the background. 

rOO 1 4 91 rO O 1 48] In derived embodiments, F(Li) can depend on a priori data 

fi'om spectral or spatiospectral libraries. 

[00150i r001491 By defining the interest factor appropriately, one can thus 

cover a range of different applications. In a preferred embodiment, the interest factor 
definitions can be pre-stored so a user can analyze a set of data using different interest 
factors. 

r00151i r001501 i t is also clear that, in the case of Walsh functions, because 
of the multi-scale nature of the Walsh pattems, one can combine raster and Walsh- 
Hadamard scanning (raster scanning at large scales, and using Walsh-Hadamard to get 
extra signal to noise ratio at fine scales, where it is needed most). This allows one to 
operate within the linear range of the detector. 

[001 52H '001 5 1 ] A lso, one can used the combined rasterAValsh idea in 
variations of the Multi-scale search and tracking algorithms. For this, whenever one is 
studying the values of a sensor associated with the sub-rectangles of a bigger rectangle, 
one could use the Walsh pattems at the relevant scale, instead of scanning the pixels at. 
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that scale. This will provide for an improvement in SNR. One could again do this only at 
finer scales, to stay in the detectors linearity range. 

rOO 1 53 1 rOO 1 521 Several signal processing tasks, such as filtering, signal 

enhancement, feature extraction, data compression and others can be implemented 
efficiently by using the basic ideas underlying the present invention. The concept is first 
illustrated in the context of one-dimensional arrays for Hadamard spectroscopy and is 
then extended to hyper-spectral imaging and various active illumination modes. The 
interested reader is directed to the book "Hadamard Transform Optics" by Martin Harwit, 
et al., published by Academic Press in 1979, which provides an excellent overview of the 
applied mathematical theory and the degree to which common optical components can be 
used in Hadamard spectroscopy and imaging applications. 

[00 151] [00 1 531 Hadamard processing refers generally to analysis tools in 

which a signal is processed by correlating it with strings of 0 and 1 (or +/- 1). Such 
processing does not require the signal to be converted firom analogue to digital, but 
permits direct processing on the analogue data by means of an array of switches 
(synapse). In a preferred embodiment of the invention, an array of switches, such as a 
DMA, is used to provide spatio-spectral tags to different radiation components. In 
alternative embodiments it can also be used to impinge spatio/spectral signatures, which 
directly correlate to desired features. 

[00155] [00154] A simple way to explain Hadamard spectroscopy is to 

consider the example of the weighing schemes for a chemical scale. Assume that we 
need to weigh eight objects, xi, X2,..., xs, on a scale. One could weigh each object 
separately in a process analogous to performing a raster scan, or balance two groups of 
four objects. Selecting the second approach, eissuming that the first four objects are in 
one group, and the second four in a second group, balancing the two groups can be 
represented mathematically using the expression: 

m = Xi+ X2 +X3 + X4 - (X5 +X6 + X7+ Xs) = (X, W), 

where x is a vector, the components of which correspond to the ordered objects Xi, 
= (1,1,1,1,-1,-1,-1,-1) and (x, w) designates the inner product of the two vectors. Various 
other combinations of object groups can be obtained and mathematically expressed as the 
inner product of the vector x and a vector of weights w, which has four +1 and four -1 
elements. 
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fflOj^r001551 F or example, w = (1, -1, 1, 1, -1,-1,1,-1) indicates that 
xi,X3,X4,X7 are on the left scale while X2 xs X6 xg are on the right. The inner product, or 
weight M = (x, w) is given by expression: 

m = (X,W) = Xi- X2 + X3 + X4 - X5 - X6 + X7 - Xg. 

r00157] |^001561 I t is well known that if one picks eight mutually orthogonal 
vectors w, which correspond, for example, to the eight Walsh patterns, one can recover 
the weight x; of each object via the orthogonal expansion method 

X = [(X, Wi) Wi -h (X, W2) W2 + . . .+ (x, Wg) Wg], 

or in matrix notation 

[W]x = m; X = [W]-^ m 
where [W] is the matrix of orthogonal vectors, m is the vector of measurements, and 
[W]-^ is the inverse of matrix [W]. 

r00158i r001571 It is well known that the advantage of using the method is 
its higher-accuracy, more precisely if the error for weighing measurement is e, the 
expected error for the result calculated from the combined measurements is reduced by 
the square root of the number of samples. This result was proved by Hotteling to provide 
the best reduction possible for a given number of measurements. 

r001591 [0Q1581 In accordance with the present invention, this signal 
processing technique finds simple and effective practical application in spectroscopy, if 
we consider a spectrometer with two detectors (replacing the two arms of the scales). 
With reference to Fig. 27, the diffraction grating sends different spectral lines into an 
eight mirror array, which redistributes the energy to the 2 detectors in accordance with a 
given pattern of +1/ -1 weights, i.e., Wi = (1,-1,1,1,-1-1,1,-1) Following the above analogy, 
the difference between the output values of the detectors corresponds to the inner product 
m = (x,W|). If one is to redistribute the input spectrum energy to the 2 spectrometers 
using eight orthogonal vectors of weights, (following the pattern by alternating the mirror 
pattems to get eight orthogonal configurations), an accurate measurement of the source 
spectram can be obtained. This processing method has certain advantages to the raster 
scan in which the detector measures one band at a time. 

[00 1 601 [001 59] Clearlv. for practical applications a precision requiring 

hundreds of bands can be required to obtain accurate chemical discrimination. However, 
it should be apparent that if once knows in advance which bands are needed to 
discriminate two compounds, the turning of the mirrors to only detect these bands could 
provide such discrimination with a single measurement. 
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[00 1611 rOO 1 601 Following is a description of a method for selecting 

efficient mirror settings to achieve discrimination using a minimum nimiber of 
measurements. In matrix terminology, the task is to determine a minimum set of 
orthogonal vectors. 

r001621 f001611 In accordance with the present invention, to this end one 
can use the Walsh-Hadamard Wavelet packets library. As known, these are rich 
collections of ±1, 0 patterns which will be used as elementary analysis patterns for 
discrimination. They are generated recursively as one follows: (a) first, double the size of 
the pattern w in two ways either as (w,w) or as (w,-w). It is clear that if various n patterns 
Wi of length n are orthogonal, then the 2n pattems of length 2n are also orthogonal. This 
is the simplest way to generate Hadamard- Walsh matrices. 

fOO 1631 TOO 1 621 The wavelet packet library consists of all sequences of 
length N having broken up in 2"" blocks, all except one are 0 and one block is filled with a 
Walsh pattern (of ± 1) of length 2^ where 1 + m = n. As known, a Walsh packet is a 
localized Walsh string of ±1. Fig. 28 illustrates all 24 library elements for N = 8. 

[0016 4 i rQ01631 A correlation of a vector x with a Walsh packet measures a 
variability of x at the location where the packet oscillates. The Walsh packet library is a 
simple and computationally efficient analytic tool allowing sophisticated discrimination 
with simple binary operations. It can be noted that in fact, it is precisely the analog of the 
windowed Fourier transform for binary arithmetic. 

r00165i r001641 As an illustration, imagine two compounds A and B with 

subtle differences in their spectrum. The task is to discriminate among them in a noisy 
environment and design efficient mirror configurations for DMA spectroscope. In 
accordance with a preferred embodiment, the following procedure can be used: 

(1) Collect samples for both A and B, the number of samples collected should 
be representative of the inherent variability of the measurements. A sample in this 
context is a full set x of the spectrum of the compound. 

(2) Compute the inner product (x, w) for all samples X of A and (y, w) for all 
samples Y of B for each fixed Walsh product w. 

(3) Measure the discrimination power pw of the pattern w to distinguish 
between compound A and B. This could be done by comparing the distribution of the 
numbers {(x, w)} to the distribution of the numbers {(y, w)}, where the farther apart 
these distributions, the better they can be distinguished. 

(4) Select an orthogonal basis of pattems w maximizing the total 
discrimination power and order them in decreasing order. 
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(5) Pick the top few patterns as an in put to a multidimensional discrimination 
method. 

[00166] r001651 As an additional optional step in the above procedure, 

experiments can be run using data on which top few selected patterns failed, and repeat 
steps 3, 4 and 5. 

rOO 1 671 rOO 1 661 B ecause of the recursive structure of the W-packet library, 
it is possible to achieve 2+3+4 in Nlog2 N computations per sample vector of length N, 
i.e. essentially at the rate data collection. It should be noted that this procedure of basis 
selection for discrimination can also be used to enhance a variety of other signal 
processing tasks, such as data compression, empirical regression and prediction, adaptive 
filter design and others. It allows to define a simple orthogonal transform into more 
usefiil representations of the raw data. Further examples are considered below and 
illustrated herein, such as the wheat protein example. 

[00 1 681 TOO 1 671 The use of Hadamard processing was considered herein to 

provide simple, computationally efficient and robust signal processing. In accordance 
with the present invention, the concept of using multiple sensors and/or detectors can be 
generalized to what is known as hyper-spectral processing. 

r00169] [00168] A s known, current spectroscopic devices can be defined 
broadly into two categories - point spectroscopy and hyper-spectral imaging. Point 
spectroscopy in general involves a single sensor measuring the electro-magnetic spectrum 
of a single sample (spatial point). This measurement is repeated to provide a point-by- 
point scan of a scene of interest. In contrast, hyper-spectral imaging generally uses an 
array of sensors and associated detectors. Each sensor corresponds to the pixel locations 
of an image and measures a multitude of spectral bands. The objective of this imaging is 
to obtain a sequence of images, one for each spectral band. At present, true hyper- 
spectral imaging devices, having the ability to collect and process the full combination of 
spectral and spatial data are not really practical as they require significant storage space 
and computational power. 

rOO 1701 [001691 In accordance with the present invention, significant 
improvement over the prior art can be achieved using hyper-spectral processing that 
focuses of predefined characteristics of the data. For example, in many cases only a few 
particular spectral lines or bands out of the whole data space are required to discriminate 
one substance over another. It is also often the case that target samples do not possess 
very strong or sharp spectral lines, so it can not be necessary to use strong or sharp bands 
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in the detection process. A selection of relatively broad bands can be sufficient to 
discriminate between the target object and the background. It should be apparent that the 
ease with which different spatio-spectral bands can be selected and processed in 
accordance with the present invention is ideally suited for such hyperspectral 
applications. A generalized block diagram of hyper-spectral processing in accordance 
with the invention is shown in Fig. 29. Fig. 30 illustrates two spectral components (red 
and green) of a data cube produced by imaging the same object in different spectral 
bands. It is quite clear that different images contain completely different kinds of 
information about the object. The same idea is illustrated in Figs. 31 and 32, where Fig. 
3 1 illustrat e s hyp e r sp e ctral imaging from airborn e camera and show s how one can 
id e ntify different crops in a scene, based on th e pr e dominant spectral charact e ristic of th e 
crop. Fig. 32 is an illustration of a hyper spectral image of human skin with spectrum 
progr e ssing from l e ft to right and top to bottom, with incr e asing wav e l e ngth. 

rOO 1711 rOO 1 701 F i gs. 33A-E illustrate different embodiments of an imaging 
spectrograph in de-dispersive mode, that can be used in accordance with this invention for 
hyper-spectral imaging in the UV, visual, near infrared and infrared portions of the 
spectrum. For illustration purposes, the figures show a fiber optic probe head with a fixed 
nvimber of optical fibers. As shown, the fiber optic is placed at an exit slit. It will be 
apparent that a multitude of fiber optic elements and detectors can be used in alternate 
embodiments. 

[00 1 721 [0 0 171 ] FIG. 34 shows an axial and cross-sectional view of the fiber 

optic assembly illustrated in Figs. 33A-E . 

[001 73] [001 72] FIG. 35 shows a physical arrangement of the fiber optic 
cable, detector and the slit. FIG. 36 illustrates a fiber optic surface contact probe head 
abutting tissue to be examined. 

ffl04^r001731 F ig. 37A and 37B illustrate a fiber optic e-Probe for pierced 
ears that can be used for medical monitoring applications in accordance with the present 
invention. 

fflO4^r001741 F ig. 38A, 38B and 38C illustrate different configurations of 
a hyper-spectral adaptive wavelength advanced illuminating imaging spectrograph 
(HAWAIIS). 

r00176i r001751 In Fig. 3 8 A, DMD (shown illuminating the -1 order) is a 

programmable spatial light modulator that is used to select spatio/spectral components 
falling upon and projecting from the combined entrance/exit slit. The illumination is fiiUy 
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programmable and can be modulated by any contiguous or non-contiguous combination 
at up to 50KHz,. The corresponding spatial resolution element located at the 
Object/sample is thus illuminated and is simultaneously spectrally imaged by the CCD 
(located in order +1 with efficiency at 80%) as in typical CCD imaging spectrographs 
used for Raman spectral imaging. 

fQQj^r001761 With reference to Figs. 38, the output of a broadband light 
source such as a TQH light bulb(lOOl) is collected by a collection optic (lens 1002) and 
directed to a spatial light modulator such as the DMA used in this example(1003). 
Specific spatial resolution elements are selected by computer controlled DMA driver to 
propagate to the transmission diffraction grating(1005) via optic (lens 1004). The 
DMA(1003) shown illuminating the -1 order of the transmission diffraction grating(1005) 
is a programmable spatial light modulator that is used to select spatio/spectral resolution 
elements projecting through the entrance/exit slit(#1007) collected and focused upon the 
sample(1009) by optic (lens 1008). The spatio/spectral resolution elements illuminating 
the sample are fully programmable. The sample is thus illuminated with specific and 
known spectral resolution elements. The reflected spectral resolution elements from 
specific spatial coordinates at the sample plane are then collected and focused back 
through the entrance/exit slit by optic (lens 1008). Optic (lens 1006) coUimates the 
returned energy and presents it to the transmission diffraction grating(1005). The light is 
then diffracted preferentially into the +1 order and is subsequently collected and focused 
by the optic (lens 1010) onto a 2D detector array(101 1). This conjugate spectral imaging 
device has the advantage of rejecting out of focus photons from the sample. Spectral 
resolution elements absorbed or reflected are measured with spatial specificity by the 
device. 

fmvm\00\77^ F igs. 45-49(A-D) illustrate hyperspectral processing in 
accordance with the present invention, including data maps, encodement mask, DMA 
programmable resolution using different numbers of mirrors and several encodegrams. 

rOO 1 791 rOO 1 781 One eispect of the present invention is the use of modulation 
of single array elements or groups of array elements to "tag" radiation impinging on these 
elements with its own pattern of modulation. In essence, this aspect of the invention 
allows to combine data from a large number of array elements into a few processing 
channels, possibly a single channel, without losing the identity of the source and/or the 
spatial or spectral distribution of the data. 
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rOO 1801 TOO 1791 A s known in the art, combination of different processing 
channels into a smaller number of channels is done using signal multiplexing. In 
accordance with the present invention, multiplexing of radiation components which have 
been "tagged" or in some way encoded to retain the identity of their source, is critical in 
various processing tasks, and in particular enables simple, robust implementations of 
practical devices. Thus, for example, in accordance with the principles of the present 
invention, using a micro mirror array, an optical router, an on-off switch (such as an LCD 
screen), enables simplified and robust image formation with a single detector and further 
makes possible increasing the resolution of a small array of sensors to any desired size. 

[00181] fQ01801 In accordance with this invention, methods for digitally- 

controlled modulation of sensor arrays are used to perform signal processing tasks while 
collecting data. Thus, the combination and binning of a plurality of radiation sources is 
manipulated in accordance with this invention to perform calculations on the analog data, 
which is traditionally done in the digital data analysis process. As a result, a whole 
processing step can be eliminated by preselecting the switching modulation to perform 
the processing before the A/D conversion, thereby only converting data quantities of 
interest. This aspect of the present invention enables realtime representation of the final 
processed data, which in processing intense applications can be critical. 

[00182] f 00181] Bv modulating the SLM array used in accordance with this 

invention, so as to compute inner products with elements of an orthogonal basis, the raw 
data can be converted directly on the sensor to provide the data in transform coordinates, 
such as Fourier transform, Wavelet transform, Hadamard, and others. This is because the 
amount of data collected is so large that it can swamp the processor or result in 
insufficient bandwidth for storage and transmission. As known in the art, without some 
compression an imaging device can become useless. As noted above, for hyper-spectral 
imaging a full spectrum (a few hundred data points) is collected for each individual pixel 
resulting in a data glut. Thus, compression and feature extraction are essential to enable a 
meaningful image display. It will be appreciated that the resulting data file is typically 
much smaller, providing significant savings in both storage and processing requirements. 
A simple example is the block 8x8 Walsh expansion, which is automatically computed by 
appropriate mirror modulation, the data measured is the actual compressed parameters. 

["0Q1831 [Q0182] In another related aspect of the present invention, data 

compression can also be achieved by building an orthogonal basis of functions. In a 
preferred embodiment, this can be achieved by use of the best basis algorithm. See, for 
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example, Coifman, R. R, and Wickerhauser, M. V., "Entropy-based Algorithms for Best 
Basis Selection", IEEE Trans. Info. Theory 38 (1992), 713-718, and U.S. Pat Nos. 
5,526,299 and 5,384,725 to one of the inventors of this application. The referenced 
patents and publications are incorporated herein by reference. 

[00183] By means of background, it is known that the reduction of 

dimensionality of a set of data vectors can be accomplished using the projection of such a 
set of vectors onto a orthogonal set of functions, which are localized in time and 
frequency. In a preferred embodiment, the projections are defined as correlation of the 
data vectors with the set of discretized re-scaled Walsh functions, but any set of 
appropriate functions can be used instead, if necessary. 

[00184] The best basis algorithm to one of the co-inventors of this 

application provides a fast selection of an adapted representation for a signal chosen from 
a large library of orthonormal bases. ExEunples of such libraries are the local 
trigonometric bases and wavelet packet bases, both of which consist of waveforms 
localized in time and frequency. An orthonormal basis in this setting corresponds to a 
tiling of the time-frequency plane by rectangles of area one, but an arbitrary such tiling in 
general does not correspond to an orthonormal basis. Only in the case of the Haar 
wavelet packets is there a basis for every tiling, and a fast algorithm to find that basis is 
known. See, Thiele, C. and Villemoes, L., "A Fast Algorithm for Adapted Time- 
Frequency Tilings", Applied and Computational Harmonic Analysis 3 (1996), 91-99, 
which is incorporated by reference. 

[00185] Walsh packet analysis is a robust, fast, adaptable, and accurate 

alternative to traditional chemometric practice. Selection of features for regression via 
this method reduces the problems of instability inherent in standard methods, and 
provides a means for simultaneously optimizing and automating model calibration. 

fQO4^r001861 The Walsh system {W„ }^^o is defined recursively by 

W2n{t) = W„(2t) + (- 1 Wn(2t - 1 ) 
W2n^i{t) = Wn (20 - (ArW„{2tA) 
^mi^\00lS^^ with Wo(t) =1 on O < t <1. if [O,l]x[0,oo] is the time 
frequency plane, dyadic rectangles are subsets of the form 

/X tt) = [T^ K2'^ (k + l)]x[2'" n,2'" (« + 1)], 
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with j, k, m and n non-negative integers, and the tiles are the rectangles of area one (}=m). 
A tile p is associated with a rescaled Walsh function by the expression 

ffiat«94r001881 F act! The function Wp and Wq are orthogonal if and only if 
the tiles p and q are disjoint. Thus, any disjoint tiling will give rise to an orthonormal 
basis of L\0,1) consisting of rescaled Walsh functions. For any tiling B, we may 
represent a function f as 

peB 

psB 

and may find an optimal such representation for a given additive cost functional by 
choosing a tiling minimizing the cost evaluated on the expansion coefficients. 

[00190] [00189] An example contrasting the use of adaptive Walsh packet 
methods with standard chemometrics for determining protein concentration in wheat is 
discussed herein. The data consists of two groups of wheat spectra, a calibration set with 
50 samples and a validation set of 54 samples. Each individual spectrum is given in units 
of log(l/R) where R is the reflectance and is measured at 1011 wavelengths, uniformly 
spaced from 1001 nm to 2617 nm. Standard chemometric practice involves computing 
derivative-like quantities at some or all wavelengths and building a calibration model 
from this data using least squares or partial least squares regression. 

fQOm4r001901 To illustrate this, let Yj be the percent protein for the i-th 

calibration spectrum S;, and define the feature Xi to be 

^ _ 5,(2182nm)->y,(2134wm) r001921 ) 
' ~ 5,(2183nm)-5,(2160«m) 



where Si(WLnm) is log(l/R) for the i-th spectrum at wavelength WL in nanometers. This 
feature makes use of 4 of the 1011 pieces of spectral data, and may be considered an 
approximate ratio of derivatives. Least squares provides a linear model AXj + B jdelding 
a prediction 7, of 7,. An estimate of the average percentage regression error is given by: 

Y-Y. 
100 



60203834.1 



41 



PSS 212.2 US 



with N being the number of sample spectra in the given data set (N is 50 for the 
caHbration set). Retaining the same notation as for the calibration set, one can compute 
the feature Xj for each validation spectmm Si and use the above model to predict Yj for 
the validation spectra. The average percentage regression error on the validation set is .62 
%, and this serves as the measure of success for the model. This model is known to be 
state-of-the-art in terms of both concept and performance for this data, and will be used as 
point of comparison. 

|"00193] [ 001911 The wavelength-by-wavelength data of each spectrum is a 

presentation of the data in a particular coordinate system. Walsh packet analysis provides 
a wealth of alternative coordinate systems in which to view the data. In such a coordinate 
system, the coordinates of an individual spectrum would be the correlation of the 
spectrum with a given Walsh packet. The Walsh packets themselves are functions taking 
on the values 1,-1, and 0 in particular patterns, providing a square- wave analogue of local 
sine and cosine expansions. Examples of Walsh packets are shown in Fig. 28. 

rOOl 9 4 1 TOO 1921 In accordance with the present invention, such functions 
can be grouped together to form independent coordinate systems in different ways. In 
particular, the Walsh packet construction is dyadic in nature and yields functions having 
N = 2*^ sample values. For N = 1024, the closest value of N for the example case of 
spectra having 1011 sample values, the number of different coordinate systems is 
approximately 10^^^. If each individual Walsh packet is assigned a numeric cost (with 
some restrictions), a fast search algorithm exists, which will find the coordinate system of 
minimal (summed) cost out of all possible Walsh coordinate systems. Despite the large 
range for the search, the algorithm is not approximate, and provides a powerful tool for 
finding representations adapted to specific tasks. 

r00195] [001931 These ideas can be applied to the case of regression for the 
wheat data in question. Any Walsh packet provides a feature, not unlike the Xi computed 
above, simply by correlating the Walsh packet with each of the spectra. These 
correlations can be used to perform a linear regression to predict the protein 
concentration. The regression error can be used as a measure of the cost of the Walsh 
packet. A good coordinate system for performing regression is then one in which the 
cost, i.e. the regression error, is minimal. The fast algorithm mentioned above gives us 
the optimal such representation, and a regression model can be developed out of the best 
K (by cost) of the coordinates selected. 
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[00 1961 TOO 1941 I n a particular embodiment, for each of the calibration 
spectra S;, first compute all possible Walsh packet features and then determine the linear 
regression error in predicting the Y; for each Walsh packet. Using this error as a cost 
measure, select a coordinate system optimized for regression, to provide a (sorted) set of 
features {Xi(l), Xi(K)} associated with each spectrum Sj. These features are 
coordinates used to represent the original data, in the same way that the wavelength data 
itself does. Four features were used in the standard model described above, and, hence, 
one can choose K = 4 and use partial least squares regression to build a model for 
predicting Yj. The average percentage regression error of this model on the validation 
data set is .7 %, and this decreases to .6 % for K = 10. Fig. 41 A shows a typical wheat 
spectrum together with one of the top 4 Walsh packets used in this model. The feature 
that is input to the regression model is the correlation of the Walsh packet with the wheat 
spectmm. (In this case the Walsh feature computes a second derivative, which suppresses 
the background and detects the curvature of the hidden protein spectrum in this region). 

rOO 1 97] f 00 1 951 Similar performance is achieved by Walsh packet analysis 

using the same number of features. The benefit of using the latter becomes clear if noise 
is taken into account. Consider the following simple and natural experiment: add small 
amounts of Gaussian white noise to the spectra and repeat the calibrations done above 
using both the standard model and the Walsh packet model. The results of this 
experiment are shown in Figure 43 A, which plots the regression error versus the 
percentage noise energy for both models (we show both the K = 4 and the K = 10 model 
for the Walsh packet case to emphasize their similarity). A very small amount of noise 
takes the two models from being essentially equivalent to wildly different, with the 
standard model having more than three times the percentage error as the Walsh packet 
model. The source of this instability for the standard model is clear. The features used in 
building the regression model are isolated wavelengths, and the addition of even a small 
amount of noise will perturb those features significantly. The advantage of the Walsh 
packet model is clear in Figure 44. The feature being measured is a sum fi-om many 
wavelengths, naturally reducing the effect of the noise. 

fmPm\00\96^ The Walsh packet method described here has other 
advantages, such as automation. The fast search algorithm automatically selects the best 
Walsh packets for performing the regression. If the data set were changed to, say, blood 
samples and concentrations of various analytes, the same algorithm would apply off the 
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shelf in determining optimal features. The standard model would need to start from 
scratch in determining via lengthy experiment which wavelengths were most relevant. 

fOQ^^rOOl 971 A daptability is also a benefit. The optimality of the 
features chosen is based on a numeric cost function, in this case a linear regression error. 
However, many cost functions can be used and in each case a representation adapted to an 
associated task will be chosen. Optimal coordinates can be chosen for classification, 
compression, clustering, non-linear regression, and other tasks, hi each case, automated 
feature selection chooses a robust set of new coordinates adapted to the job in question. 

["00200] [00198] In accordance with an embodiment of the present invention, 

a system in which a video camera is synchronized to the tunable light source modulation 
allowing analysis of the encoded spectral bands from a plurality of video images, thereby 
providing a multispectral image. Since the ambient light is not modulated it can be 
separated from the desired spectral information. This system is the functional equivalent 
of imaging the scene a number of times with a multiplicity of color filters. It allows the 
formation of any virtual photographic color filter with any absorption spectrum desired. 
A composite image combining any of these spectral bands can be formed to achieve a 
variety of image analysis, filtering and enhancing effects. 

[0020 1 1 TOO 1 99] For example, an object with characteristic spectral signature 
can be highlighted by building a virtual filter transparent to this signature and not to 
others (which should be suppressed). In particular, for seeing the concentration of protein 
in a wheat grain pile (the example discussed below) it would be enough to illuminate with 
two different combination of bands in sequence and take the difference of the two 
consecutive images. More elaborate encodements can be necessary if more spectral 
combinations has to be measured independently, but the general principle remains. 

r00202i r002001 in a different embodiment, an ordinary video camera used 
in accordance with this invention is equipped with a synchronized tunable light source so 
that odd fields are illuminated with a spectral signature which is modulated from odd field 
to odd field while the even fields are modulated with the complementary spectral 
signature so that the combined even odd light is white. Such an illumination system 
allows ordinary video imaging which after digital demodulation provides detailed spectral 
information on the scene with the same capabilities as the gray level camera. 

[00203] [ 00201 1 This illumination processing system can be used for 

machine vision for tracking objects and anywhere that specific real time spectral 
information is useful 



60203834.1 



44 



PSS 212.2 US 



[0020 4 1 r002021 In another embodiment, a gray level camera can measure 
several preselected light bands using, for example, 16 bands by illuminating the scene 
consecutively by the 16 bands and measuring one band at a time. A better result in 
accordance with this invention can be obtained by selecting 16 modulations, one for each 
band, and illuminating simultaneously the scene with all 16 colors. The sequence of 16 
frames can be used to demultiplex the images. The advantages of multiplexing will be 
appreciated by those of skill in the art, and include: better signal to noise ratio, 
elimination of ambient light interference, tunability to sensor dynamic range constraints, 
etc. 

r00205i r002031 A straightforward extension of this idea is the use of this 
approach for multiplexing a low resolution sensor array to obtain better image quality. 
For example, a 4X4 array of mirrors with Hadamard coding could distribute a scene of 
400x400 pixels on a CCD array of 100X100 pixels resulting in an effective array with 16 
times the number of CCD. Further, the error could be reduced by a factor of four over a 
raster scan of 16 scenes. 

[002061 [002041 In accordance with the present invention by irradiating a 
sample of material with well-chosen bands of radiation that are separately identifiable 
using modulation, one can directly measure constituents in the material of interest. This 
measurement, for example, could be of the protein quantity in a wheat pile, different 
chemical compounds in human blood, or others. It should be apparent that there is no real 
limitation on the type of measurements that can be performed, although the sensors, 
detectors and other specific components of the device, or its spectrum range can differ. 

[00207] [ 002051 In the following example we illustrate the measurement of 
protein in wheat. The data consists of two groups of wheat spectra, a calibration set with 
50 samples and a validation set of 54 samples. 

fQQ2O84[002061 Fig. 39 shows a DMA search by splitting the scene. The 
detection is achieved by combining all photons from the scene into a single detector, then 
splitting the scene in parts to achieve good localization. In this example, one is looking 
for a signal with energy in the red and blue bands. Spectrometer with two detectors, as 
shown in Fig. 27 can be used, so that the blue light goes to the top region of the DMA, 
while the red goes to the bottom. 

[00209] [00207] First, the algorithm checks if it is present in the whole scene 
by collecting all photons into the spectrometer, which looks for the presence of the 
spectral energies. Once the particular spectrum band is detected, the scene is split into 
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four quarters and each is analyzed for presence of target. The procedure continues until 
the target is detected. 

fflQa4mr002081 F ig. 40 illustrates the sum of wheat spectra training data 
(top) Sum of |w| for top 10 wavelet packets (middle) and an example of protein spectra - 
soy protein (bottom). The goal is to estimate the amount of protein present in wheat. The 
middle portion of the figure shows the region where the Walsh packets provide useful 
parameters for chemo-metric estimation. 

r00211i r002091 Fig. 11 illustratos Figs. 41-1 and 41-2 illustrate the top 10 

wavelet packets in local regression basis selected using 50 training samples. Each Walsh 
packet provides a measurement useful for estimation. For example, the top line indicates 
that by combining the two narrow bands at the ends and then subtracting the middle band 
we get a quantity which is linearly related to the protein concentration. Fig. 42 is a scatter 
plot of protein content (test data) vs. correlation with top wavelet packet. This illustrates 
a simple mechanism to directly measure relative concentration of desired ingredients of a 
mixture. 

r002121 |'002101 It will be appreciated that in this case one could use an 
LED-based flashlight illuminating in the three bands with a modulated light, which is 
then imaged with a CCD video camera that converts any group of consecutive three 
images into an image of protein concentration. Another implementation is to replace the 
RGB filters on a video camera by three filters corresponding to the protein bands, to be 
displayed after subtraction as false RGB. Various other altemative exist and will be 
appreciated by those of skill in the art. 

rOQ213] [0021 1] F ig 43 illustrates PLS regression of protein content of test 

data: using top 10 wavelet packets (in green - 1.87% error, fi-om 6 LVs) and top 100 (in 
red - 1.54% error fi-om 2 LVs) - compare with error of 1.62% from 14 LVs using all 
original data. This graph compares the performance of the simple method described 
above to the true concentration values. 

fQQ2j-44rQ02121 F ig 44 illustrates the advantage of DNA-based Hadamard 
Spectroscopy in terms of visible improvement in the SNR of the signal for the Hadamard 
Encoding over the regular raster scan. 

rQ0215i r002131 It will be appreciated that the above approach can be 

generalized to a method of detecting a chemical compound with known absorption lines. 
In particular, a simple detection mechanism for compounds with known absorption is to 
use an active illumination system that transmits light (radiation) only in areas of the 
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absorption spectrum of the compound. The resulting reflected light will be weakest 
where the compound is present, resulting in dark shadows in the image (after processing 
away ambient light by, for example, subtracting the image before illumination). Clearly, 
this approach can be used to dynamically track objects in a video scene. For example, a 
red ball could be tracked in a video sequence having many other red objects, simply by 
characterizing the red signature of the ball, and tuning the illumination to it, or by 
processing the refined color discrimination. Clearly this capability is usefiil for 
interactive TV or video-gaming, machine vision, medical diagnostics, or other related 
applications. Naturally, similar processing can be applied in the infrared range (or UV) to 
be combined with infrared cameras to obtain a broad variety of color night vision or (heat 
vision), tuned to specific imaging tasks. To encode the received spatial radiation 
components one can use pulse code modulation (PCM), pulse width modulation (PWM), 
time division multiplexing (TDM) and any other modulation technique that has the 
property of identifying specific elements of a complex signal or image. 

r00216i r002141 In accordance with the invention, in particular applications 

one can rapidly switch between the tuned light and its complement, arranging that the 
difference will display the analate of interest with the highest contrast. In addition, it is 
noted that the analate of interest will flicker, enabling detection by the eye. Applications 
of this approach in cancer detection in vivo, on operating table, can easily be foreseen. 

r002 1 71 [002 1 51 Another straightforward extension of the present invention 

is method for initiating select chemical reactions using a tunable light source. In 
accordance with this aspect, the tunable light source of this invention can be tuned to the 
absorption profile of a compound that is activated by absorbing energy, to achieve curing, 
drying, heating, cooking of specific compounds in a mixture. Applications further 
include photodynamic therapy, such as used in jaundice treatment, chemotherapy, and 
others. 

r002 1 8 1 r002 1 61 Y et another application is a method for conducting 
spectroscopy with determining the contribution of individual radiation components from 
multiplexed measurements of encoded spatio-spectral components. In particular a 
multiplicity of coded light in the UV band could be used to cause fluorescence of 
biological materials, the fluorescent effect can be analj^ed to relate to the specific coded 
UV fi-equency allowing a multiplicity of measurements to occur in a multiplexed form. 
An illumination spectrum can be designed to dynamically stimulate the material to 
produce a detectable characteristic signature, including fluorescence effects and multiple 
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fluorescent effects, as well as Raman and polarization effects. Shining UV light in 
various selected wavelengths is known to provoke characteristic fluorescence, which 
when spectrally analyzed can be used to discriminate between various categories of living 
or dead cells. 

ffiQ2j-9^r002 1 71 Another application of the system and method of this 
invention is the use of the OSPU as a correlator or mask in an optical computation device. 
For example, an SLM, such as DMA can act as a spatial filter or mask placed at the focal 
length of a lens or set of lenses. As illustrated above, the SLM can be configured to reject 
specific spatial resolution elements, so that the subsequent image has properties that are 
consistent with the spatial filtering in Fourier space. It will be apparent that the transform 
of the image by optical means is spatially effected, and that the spatial resolution of 
images produced in this manner can be altered in any desired way. 

r00220i r002181 Y et another area of use is performing certain signal 
processing functions in analog domain. For example, spatial processing with a DMA can 
be achieved directly in order to acquire various combinations of spatial patterns. Thus, an 
array of mirrors can be arranged to have all mirrors of the center of the image point to one 
detector, while all the periphery goes to the other. Another usefiil arrangement designed 
to detect vertical edges will raster scan a group of, for example, 2x2 mirrors pointing left 
combined with an adjacent group of 2x2 mirrors pointing right. This corresponds to a 
convolution of the image with an edge detector. The ability to design filters made out of 
pattems of 0,1,-1 i.e., mirror configurations, will enable the imaging device to only 
measure those features which are most usefiil for display, discrimination or identification 
of spatial pattems. 

r0022 1 1 r002 1 91 The design of filters can be done empirically by using the 
automatic best basis algorithms for discrimination, discussed above, which is achieved by 
collecting data for a class of objects needing detection, and processing all filters in the 
Walsh Hadamard Library of wavelet packets for optimal discrimination value. The 
offline default filters can then be upgraded online in real-time to adapt to filed conditions 
and local clutter and interferences. 

ffi03334r0022Q1 Turning now to FIG. 50, there is shown a block diagram of 
a near-infi-ared spectral target identification system (NSTIS) 1 000 in accordance with an 
embodiment of the present invention. Light from a scene 1010 is imaged through fore 
optics 1020 and onto a spatial light modulator, such as a DMD 1030. This imaging can 
be accomplished, for example, using an ordinary camera lens. The light from the DMD 
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1030 is then sent through an imaging spectrograph 1040 which has been modified to have 
its slit removed and disposed such that the DMD 1030 sits in the plane in which the slit 
previously sat. The output light of the imaging spectrograph 1040 is then sent, in the 
usual manner, onto a camera 1050. 

fQQ33^r0022l1 The NSTIS system 1000 collects data by sequentially 
rendering a sequence of coded pattems on the DMD 1030, and, for each pattern, grabbing 
a frame of data from the camera 1050. In accordance with an aspect of the present 
invention, the coded pattems on the DMD 1030 consists of a series of binary images, 
each with a single column of mirrors in the "on" position, and all other mirrors in the 
"off position. The on-column is moved across the DMD 1030 (e.g., from left to right). 
In so doing, the scene 1010 is scanned. At each fixed position of the on-column, the 
system 1000 behaves like an ordinary imaging spectrograph with slit position 
corresponding to the location of the on-row. As the row is scanned across the image of 
the scene 1010, a spectrograph image is captured, one for each position of the "virtual 
slit". In this manner, the entire scene 1010 is scanned without the need for macro-moving 
parts. 

[002241 [002221 In accordance with an embodiment of the present invention, 
instead of rendering a series of virtual slit images, a series of encodements, such as a 
series of Hadamard pattems, are rendered. For example, a Hadamard pattern of columns 
can be rendered. That is, in contrast to turning the rows "on" one at a time for a total of N 
images, (N+l)/2 rows are tumed "on" at a time in N different ways to collect N images. 
The selection of which (N+l)/2 rows to turn "on" is dictated by a Hadamard code of 
length N. In this manner, a Hadamard encoded dataset is collected. It is appreciated that 
this encoded dataset is the Hadamard transform of the dataset collected with respect to the 
"virtual slit" embodiment discussed in the previous paragraph. Hence, an inverse 
Hadamard transform is only necessary to produce essentially the same data as the 'Virtual 
slit embodiment, ignoring system response and signal-to-noise-ratio (SNR) issues. 
Although these are typically important issues, and the Hadamard multiplexed use 
generally provides many advantages noted herein. 

[002251 [002231 Returning to the discussion about scanning of single virtual 
slits across an image. FIG. 59 demonstrates that as the virtual slit is scanned across the 
input image, the dispersed light from the spectrograph portion 1040 of the system 1000 
moves by a corresponding amount across the output image. Depending on the 
magnification factor of the system 1000, a shift of one pixel on the DMD 1030 can 
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correspond to a shift of other than one pixel in the output image. In accordance with an 
aspect of the present invention, a magnification factor of less than 1 can be selected and 
used, so that the mirrors of the DMD 1030 sub-modulate the pixels of the camera 1050, 
thus increasing the system resolution. 

r00226i r002241 FIG. 59 is an example of a mapping of micro-mirror 
columns to the dispersed spectral images at the focal plane of a 2D array sensor, focal 
plane array (FPA) or camera in accordance with an embodiment of the present invention. 
That is, a map of DMD SLM micro-mirrors to the FPA is depicted in FIG. 59. The DMD 
micro-mirror representation is shown on the left column of FIG. 59. The corresponding 
spectrally dispersed images of the micro-mirror column are shown as a shaded block at 
the representative FPA on the right column of FIG. 59. As the column of micro-mirrors 
that are selected to be in the "on" condition moves fi-om left to right, the spectrally 
dispersed images of the column move from left to right on the FPA for a given spectral 
range of operation. A multitude of colunms can be simultaneously selected which has the 
effect of presenting a multitude of overlapping spectral images onto the FPA 
(multiplexing). 

r00227i r002251 It is appreciated that the data collected by the hyper-spectral 
imaging system of the present invention results in a skewed datacube. That is, if one 
imagines the image frames collected as being stacked directly on top of each other, then 
the active area of data collected in the images forms a skewed parallelepiped within a 
solid rectangle corresponding to the image stack. In order to create a traditional datacube, 
with two spatial dimensions and one spectral dimension, this data must be deskewed. 
This can be accomplished by simply shifting each successive image by one more pixel 
than the previous image, and selecting a portion of the shifted image corresponding to the 
active data area, for example see FIG. 59. When the system magnification factor is such 
that a shift of one pixel on the DMD 1030 corresponds to other than a shift of one pixel 
on the camera 1050, interpolation schemes known to those of ordinary skill in the art can 
be employed to correct for the non-integer shift. 

fQQ2284rQ02261 The digital micro-mirror device (DMD) spatial light 
modulator (SLM) presents a challenge in that typical imaging optics have image planes 
that are perpendicular to the optical propagation axis Z. When the object is imaged by 
these conventional optical imaging systems onto the DMD 1030, the image becomes an 
object for the imaging spectrograph and the DMD 1030 can selectively relay spatial 
resolution elements of this object-image onto the imaging spectrograph portion 1040 of 
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the system 1000. However, this requires special optical systems capable of handling the 
tilted object. Furthermore the resultant image itself is tilted. Generally speaking, this 
results in poor optical performance for imaging systems and typical imaging systems 
cannot cope with such condition at low F numbers of operation. 

fflQ3394r002271 F IG. 51 shows an example of the tilted object plane imaged 
onto a resultant tilted image pleine using a simple image relay lens system 1100. Light 
from a scene 1110 is imaged through a fore optic or objective lens 1 120 and through a 
TIR prism 1 130 such that the light passes through the internal face 11 35 of the TIR prism 
1 130, and onto a DMD 1 140. The system 1 100 is disposed such that light from the "on" 
mirrors of the DMD 1 140 bounce off of the face 1 135 and onto the imaging spectrograph 
1 150. Whereas Hght from the "off mirrors of the DMD 1 140 pass through the face 1 135 
and do not propagate fiirther through the system 1 100 and onto the imaging spectrograph 
1 150. The imaging spectrograph 1 150 includes a relay lens 1 152 and a dispersion device 
1 154, such that the light from the "on" mirrors of the DMD 1 140 goes to the relay lens 
1152 and then onto the dispersion device 1154. The dispersion device 1154 then 
disperses the light across a focal plane array 1 160. The operation of the relay lens system 
1 100 is not described herein because its operation is similar to the NSTIS 1000 of FIG. 
50. 

fQQ2^r002281 T urning now to FIG. 52, there is illustrated an model of a 

representative tilted plane imaging spectrograph system in accordance with an 
embodiment of the present invention. The source image is coUimated and re-imaged 
without magnification after dispersion by a set of paraxial lens systems. The figure 
illustrates an example of an asymmetric imaging relay spectrograph in accordance with an 
embodiment of the present invention using simulated DMD tilted object 1171, two 
paraxial lenses 1 172, a grating 1 173 and a tilted FPA at the image plane 1 174. 

r00231i r002291 A model of a lens 1210 imaging a source onto the DMD 
SLM through a Near Infrared optimized total internal reflection (TIR) prism assembly 
1220 in accordance with an embodiment of the present invention is shown in FIG. 53. 
The imaging lens 1210 and the TIR prism assembly 1220 improves the F numbers of 
operation by sung the DMD as an SLM to select spatial resolution elements of an image 
to pass to subsequent optical elements of the imaging or spectral imaging system. That is, 
the prism assembly 1220 fiinctions to reduce effective F numbers of operation by creating 
a greater path divergence of the incoming and outgoing beam of light. The prism 
assembly 1220 can be so constructed that it fiinctions to separate both the "on" deflected 
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light (+10 degree mirror position) and the "off light (-10 degree mirror positions). FIG. 
52 shows only the "on" or (+10) degree position of the mirrors so as to direct the light to 
the subsequent optical system such as the imaging spectrograph. 

[Q0232] [00230] In accordance with an embodiment of the present invention, 
a DMD enabled spectral imaging system 1300 is shown in FIG. 54. The system 1300 
uses a typical imaging lens 1310 to present an image of the object under observation to 
the DMD SLM through a near-infrared (NIR) TIR prism assembly. The spatial resolution 
elements are selected to pass onto the relay lens system and transmission diffraction 
grating, and presented to the camera 2D sensor array or focal plane array (FPA). The 
camera lens or imaging lens receives light from the source object and re-images the object 
onto the DMD where the plane of micro-mirrors are surface normal to the optical 
propagation axis Z. The DMD can then be used to select which spatial resolution 
elements of the image will propagate through the TIR prism and into the relay lens, 
through the transmission grating and spectrally imaged onto the camera sensor or focal 
plane may (FPA). 

r00233] ['00231] In accordance with an aspect of the present invention, the 
system 1300 can comprises two lens sets that are used in conjugate positions such that the 
first lens set collimates the light from the DMD SLM to the diffraction optical element 
and the second lens focuses the dispersed light onto the FPA. The lenses can be either 
refractive elements or reflective optical systems. 

fflO234;r002321 Turning now to FIG. 55, there is a photograph of a 

prototype of a hyper-spectral imaging system in accordance with an embodiment of the 
present invention. The photograph shows an embodiment in which light from a scene of 
interest is imaged by an imaging optic 5060, through a TIR prism assembly 5040, onto a 
DMD 5050, back through the TIR prism assembly 5040, and sent onto and off of a 
grating 5030 through a series of transfer optics 5020. The resulting spatiospectral image 
is sent to and captured by a camera 5010. 

[Q0235H "00233] In accordance with an embodiment of the present invention, 
the hyper-spectral imaging system utilizes all reflective optical components such that the 
system is not limited by the spectral range of transmission of the refractive elements. The 
utilization of all reflective optical elements also permit chromatic error free operation. In 
accordance with an aspect of the present invention, the hyper-spectral imaging system 
utilizes a special optical relay system developed by Plain Sight Systems (PSS) to place 
the image onto the DMD in a manner where the image plane that is perpendicular to the 
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Optical propagation axis Z is made tilted for presentation to the DMD without a TIR 
prism assembly. 

rQ0236i r002341 ^An optimized model of an all reflective image delivery 

system for a DMD enabled SLM spectral imaging system or imaging system in 
accordance with an embodiment of the present invention is shown in FIG. 56. The object 
5100 to be imaged onto the DMD is shown at the bottom of the image perpendicular to 
the optical propagation axis. The design and placement of the relay optical system 
presents a tilted image 5120 appropriate for presentation to the DMD SLM such that the 
subsequent spatial resolution elements so selected propagate perpendicular the DMD 
micro-mirror plane. This allows conventional optical systems to be employed 
subsequently to the DMD such that image quality is maintained to a high degree of 
coherence and performance. 

r00237i r002351 T he optimized all reflective model takes an object plane 
and presents a tilted image at the correct angle to the DMD so as to allow the use of 
typical optical sub-systems to receive the spatial resolution elements that are selected to 
propagate through the system in a conventional manner. This is important because most 
spectral imaging systems are very sensitive to the degree of collimation for resolution 
elements impinging upon a diffraction element. If the DMD object that is a result of the 
primary object image is tilted, the conventional spectral imaging system has difficulty 
with spectrally re-imaging the primary object at the FPA. Beginning with an Offher Type 
1:1 all spherical image relay system, one can begin to decenter and tilt the optical system 
such that the object plane of the relay is re-imaged at a tilt with respect to the optical axis 
of propagation. This can be accomplished by setting up an optical design merit function 
in a given optical design software program such that the system is perturbed in 
accordance with the figures of merit dictating the solution. Such solution, as embodied 
for example by reflective elements 5110, 5111, and 5112, being the optimization of a 20 
degrees tilt corresponding to the angle required by the DMD to subsequently project the 
light from the object image on the DMD perpendicular to the plane of the image on the 
DMD. 

r0023 81 r0023 6] FIG. 57 illustrates an optical relay system in accordance 
with an embodiment of the present invention optimized for use with a TIR prism 
assembly and the DMD SLM such that the tilted image presented to the DMD reflects 
selected resolution elements perpendicular to the micro-mirror array plane. This allows 
conventional optical systems such as refractive coUimating lenses to be used. That is, 
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FIG. 57 shows an extension of the system shown in FIG. 56 for use with a TIR prism 
assembly. In the subassembly shown in FIG. 57, light reflected off of the activated pixels 
of a DMD 6000, is sent through a TIR prism assembly 6010, onto a series of reflective 
lenses 6020, 6021, 6022, and then onto a fold mirror 6023, in order to produce an output 
image 6030 that is tilt corrected as disclosed herein. 

fQQa3»tr002371 F IG. 58 shows a CAD drawing of NSTIS in accordance 
with an embodiment of the present invention. The optical path is shown beginning at the 
upper left with a typical refractive camera lens system that places and image at the object 
plane of the custom designed flat to tilted image relay system. The image of the scene 
from the camera lens 7010 is folded by a flat mirror 7015 to a spherical collection mirror 
7020 and onto spherical mirror 7030 and onto spherical mirror 7040 where the light is 
passed through a special total internal reflection prism designed for this application 7050 
where the image is folded onto the DMD 7060. From the DMD 7060 the light propagates 
back through 7050 and is picked up by a camera lens 7070 where the rays are coUimated 
onto a grating 7080 and subsequently re-imaged and demagnified by a camera lens 7090 
and 7100 to focus onto the InGaAs focal plane 7110. 

fQQ24mr002381 In the hyper-spectral imaging system of FIG. 58, a 
refractive camera lens receives the light from the source object and images it onto the 
object plane of the all reflective treuisfer optical system. The all reflective transfer optical 
system takes this conventional perpendicularly presented image with respect to the optical 
axis from the conventional camera lens system and presents it through the TIR prism to 
the DMD micro-mirror plane in a tilted fashion with respect to the optical axis such that 
the selected spatial resolution elements can propagate to the imaging spectrograph portion 
of the system. This condition allows more typical imaging optical systems to be used and 
improves the collimation of all of the desired spatial resolution elements used in the DMD 
SLM. Improving the collimation improves the effectiveness of the diflfractive element 
and improves the subsequent image quality as it is focused by the focusing optical 
elements in the spectrograph portion of the system. 

r002'in r002391 in accordance with an embodiment of the present invention, 
the system comprises software for acquisition of hyper-spectral data, and a user interface 
for interactively selecting regions from a 2D projection of the 3D hyper-spectral 
datacube. Preferably, the system also comprises software for computing features of the 
selected regions, and display of the 3D datacube, project into 2D via the selected features. 
For example, the software of the system can compute the mean spectral vector in each 
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region, and then compute a Gram Schmidt orthogonalization of the selected vectors. 
When there are 3 selected vectors, the output of the Gram Schmidt algorithm can be used 
to compute 3 spectral inner products over the datacube, and the results are used to render 
an RGB image of the datacube. The process can then be iterated, providing the user the 
ability to select regions in the original and the processed 2D projections. 

r00212i r002401 In accordance with an embodiment of the present invention, 
the MEMS-adapted spectrograph imaging system is used in conjunction with the tunable 
light source. Light from the tuned light is used to illuminate a scene, sample, or one or 
more materials of interest. The light reflected, scattered and/or emitted from the sample 
or materials is imaged by the MEMS-adapted spectrograph imaging system, thereby 
performing fluoresence imaging without the need for excitation and emission filters. 

r00243 1 r0024 1 1 W hile the foregoing has described and illustrated aspects of 
various embodiments of the present invention, those skilled in the art will recognize that 
alternative components and techniques, and/or combinations and permutations of the 
described components and techniques, can be substituted for, or added to, the 
embodiments described herein. It is intended, therefore, that the present invention not be 
defined by the specific embodiments described herein, but rather by the appended claims, 
which are intended to be construed in accordance with the well-settled principles of claim 
construction, including that: each claim should be given its broadest reasonable 
interpretation consistent with the specification; limitations should not be read from the 
specification or drawings into the claims; words in a claim should be given their plain, 
ordinary, and generic meaning, unless it is readily apparent from the specification that an 
unusual meaning was intended; an absence of the specific words "means for" connotes 
applicants' intent not to invoke 35 U.S.C, §1 12 (6) in construing the limitation; where the 
phrase "means for" precedes a data processing or manipulation "fianction," it is intended 
that the resulting means-plus-fimction element be construed to cover any, and all, 
computer implementation(s) of the recited "function"; a claim that contains more than one 
computer-implemented means-plus-fimction element should not be construed to require 
that each me2ms-plus-fiinction element must be a structurally distinct entity (such as a 
particular piece of hardware or block of code); rather, such claim should be construed 
merely to require that the overall combination of hardware/firmware/software which 
implements the invention must, as a whole, implement at least the fiinction(s) called for 
by the claim's means-plus-fiinction element(s). 
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